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Abstract 

This  purpose  of  this  reooareh  was  to  analyze  the 
Requirements  Correlation  Matrix  (RCM)  and  Baseline 
Correlation  Matrix  (BCM) .  The  specific  question  addressed 
by  the  research  was  "What  improvements  can  be  made  to  better 
define  and  document  the  operational  performance  requirements 
during  the  acquisition  of  aeronautical  weapon  systems?" 
Requirements  personnel  at  using  commands,  the  developing 
command,  and  the  operational  test  agency  were  surveyed  to 
determine  who  selects  the  parameters  and  values  in  RCMs  and 
BCMs ,  and  whether  these  personnel  had  differing 
interpretations  of  both  requirements  and  specifications. 

The  results  indicated  there  was  not  a  clear  understanding  of 
parameters  and  values  as  requirements  or  specifications  by 
all  requirements  personnel,  and  a  lack  of  sufficient 
acquisition  education  opportunities  existed  for  requirements 
personnel  at  the  using  commands.  The  recommendations  were 
that  requirements  personnel  must  understand  the  implications 
of  the  selection  of  parameters  in  RCMs/BCMs,  more 
requirements  oriented  acquisition  education  for  using 
command  personnel  is  necessary  to  enhance  the  effectiveness 
of  the  RCM/BCM,  and  that  a  formal  general  officer  review  of 
requirements  should  be  conducted  for  major  weapon  systems  to 
ensure  that  they  are  mission  essential. 
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AN  ANALYSIS  OF  THE 

REQUIREMENTS  CORRELATION  MATRIX  (RCM) 

AND 

BASELINE  CORRELATION  MATRIX  (BCM) 


I.  INTRODUCTION 


General  Issue 

The  purpose  of  the  Air  Force  weapon  system  acquisition 
process  is  to  develop,  produce,  test,  and  field  a  weapon 
system  that  meets  a  user's  valid  need.  "Acquisition  of  the 
tools  of  defense  is  an  immense  and  complex  enterprise" 
(23:60).  The  acquisition  of  a  major  weapon  system  usually 
involves  numerous  Air  Force  organizations,  civilian 
contractors,  and  Congress,  and  may  span  a  period  of  time  as 
long  as  ten  to  fifteen  years.  Studies  of  program  management 
both  within  and  outside  the  Department  of  Defense  (DOD)  have 
consistently  found  that  close  ties  to  the  user  are  essential 
for  successful  programs  (1;  18;  20;  23;  26).  The  Packard 
Commission  noted  that,  in  most  cases,  the  development  and 
production  of  weapon  systems  cost  too  much,  took  too  long, 
and  the  systems  did  not  perform  as  they  were  originally 
intended  (23:xxii).  The  weapon  system  process  is  influenced 
by  factors  such  as  requirements  stability,  funding 
availability  and  stability,  cost  stability,  and  schedule 
constraints.  The  definition  of  program  requirements  sets 
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the  stage  for  the  entire  program.  "How  can  the  acquisition 
process  be  improved?"  is  an  important  question  to  be 
addressed  in  an  effort  to  make  the  acquisition  process  more 
efficient  and  effective. 

Specific  Problem 

The  Air  Force  has  established  policies  and  procedures 
which  govern  the  definition  of  requirements  at  program 
inception  and  the  periodic  reviewing/updating  of  these 
requirements  as  the  program  matures.  However,  following 
these  policies  and  procedures  to  the  letter  will  not 
guarantee  success  if  the  program  requirements  are  not  stated 
in  meaningful  terms.  In  answering  the  general  question 
posed  above,  a  more  specific  question  must  be  addressed: 
"What  improvements  can  be  made  to  better  define  and  document 
the  operational  performance  requirements  associated  with  the 
acquisition  of  weapon  systems?" 

Research  Objectives 

In  addressing  this  specific  question,  the  following 
objectives  were  considered. 

1.  Identify  the  important  issues  of  the  operational 
requirements  process  within  the  Air  Force  systems 
acquisition  lifecycle. 

2.  Determine  who  selects  the  RCM/BCM  parameters  and  the 
associated  values,  which  should  identify  the  user's 
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requirements,  the  contractual  specifications,  and  the  IOT&E 
criteria  for  the  weapon  system. 

3.  Determine  if  requirements  personnel  at  using 
commands,  the  developing  command,  and  the  operational  test 
agency  have  differing  interpretations  of  both  operational 
requirements  and  specifications. 

Scope 

Initially,  this  thesis  describes  the  broad  topic  of  the 
Air  Force  acquisition  process.  A  thorough  understanding  of 
this  process  is  necessary  before  addressing  the  problem 
outlined  earlier.  This  is  accomplished  in  the  literature 
review  by  addressing  the  first  research  objective.  The 
review  of  literature  for  this  paper  consists  mainly  of 
professional  military  journals,  DoD  publications,  Air  Force 
regulations,  and  two  lessons  learned  databases.  Next,  the 
thesis  focuses  on  Air  Force  aeronautical  requirements 
personnel  at  the  using  commands  (MAC,  SAC,  and  TAC) ,  the 
developing  command  (ASD) ,  and  the  operational  test  agency 
(AFOTEC)  who  are  involved  in  the  selection  of  parameters  and 
values  for  RCMs  and  BCMs.  The  procedures  for  conducting 
this  phase  of  the  research  are  presented  in  Chapter  III, 
Methodology. 
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II.  LITERATURE  REVIEW 


Overview 

This  chapter  presents  a  review  of  the  literature  and 
provides  the  background  information  necessary  for  a  study  of 
the  selection  of  the  parameters  and  their  associated  values 
found  in  RCMs/BCMs  which  are  used  to  evaluate  aeronautical 
systems  being  acquired  through  the  Air  Force  systems 
acquisition  process.  This  was  accomplished  by  addressing 
the  first  research  objective.  Specific  Air  Force 
regulations  and  documents  were  reviewed  to  provide  an 
understanding  of  the  Air  Force  systems  acquisition  process 
at  an  important  stage:  the  selection  of  the  parameters  and 
values  for  RCMs  and  BCMs.  Additional  material  from 
professional  military  journals  and  two  Air  Force  lessons 
learned  databases  was  used  to  describe  the  merits  and 
shortcomings  that  are  inherent  in  this  process.  The 
methodology  for  answering  the  remaining  research  objectives 
is  provided  in  Chapter  III,  Methodology. 

Program  Management 

Program  management  is  generally  associated  with  the 
undertaking  of  a  complex,  nonrepetitive  task  that  occurs 
during  a  discrete  period  of  time.  It  is  distinctly 
different  from  product  management  because  the  latter 
concerns  managing  the  repetitive  production  and 
merchandising  of  an  item.  Program  management,  by 
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comparison,  is  much  more  complex  because  it  generally 
involves  the  planning,  execution  and  direction  of  an 
organization's  activities  in  the  attainment  of  its 
objectives.  Weapon  systems  acquisition  within  DoD  is  a 
process  well  suited  to  program  management.  The  objectives 
of  Air  Force  acquisition  program  management  are: 

1)  Field  the  most  cost  effective  combat  and  combat 
support  capability  that  fulfills  the  user's  need. 

2)  Field  fully  supported  systems  that  are  reliable  and 
maintainable . 

3)  Maintain  acquisition  excellence  through  innovative 
and  aggressive  approaches  addressing  user  needs. 

4)  Develop  alternatives  to  meet  the  user's  need. 

5)  In  concert  with  laboratory  or  air  logistic  center 
commanders,  support  and  pursue  technological  advances. 

6)  Balance  available  resources  against  program 
requirements  and  direction.  (2:6) 

Within  the  Air  Force,  a  program  is  managed  by  an  individual 
who  may  be  referred  to  as  a  project  manager,  program 
manager,  or  program  director.  Throughout  the  remainder  of 
this  paper,  this  individual  is  referred  to  as  the  program 
director.  The  program  director  is  charged  with  a  wide 
variety  of  responsibilities  including  developing  the 
acquisition  strategy,  providing  budgetary  estimates  and 
alternatives,  establishing  a  program  schedule,  developing 
contractual  agreements,  and  conducting  the  day-to-day 
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management  of  the  program.  The  organization  which  manages  a 
development  program  is  known  as  a  system  program  office 
(SPO) . 

Military  Program  Management 

In  July  1970,  a  Blue  Ribbon  Defense  Panel  established  by 
the  President  and  the  Secretary  of  Defense  reported  on  the 
organization  and  management  of  DoD  procurement  policies  and 
practices  (17:v).  One  of  the  findings  of  the  Panel  stated 
that  the  development  and  procurement  of  military  hardware 
was  traumatized  by  the  "now  familiar"  symptoms  of  trouble: 

1)  Major  cost  growth  or  overruns; 

2)  Schedule  slips;  and 

3)  Failures  in  performance.  (17:62) 

Sixteen  years  later,  the  Packard  Blue-Ribbon  Commission  came 
to  the  same  conclusion  that  weapon  system  acquisition 
generally  takes  too  long,  costs  too  much,  and  the  weapon 
system  does  not  perform  as  it  was  originally  intended 
(23:xxii).  The  problems  of  program  management  are  not 
limited  to  DoD.  A  well-respected  book  by  Thomas  J.  Peters 
and  Robert  H.  Waterman,  In  Search  of  Excellence:  Lessons 
from  America's  Best-Run  Companies,  provides  insight  into  the 
attributes  of  a  successful  company  or  organization.  As 
defined  by  Peters  and  Waterman,  an  "excellent  company"  must 
thoroughly  understand  its  objectives  (19:26).  This 
philosophy  is  easily  translated  to  Air  Force  program 
management . 
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A  study  of  commercial  programs  by  the  Packard  Blue-Ribbon 
Commission  identified  six  common  characteristics  which  were 
present  in  successful  programs: 

1)  Clear  command  channels; 

2)  Program  stability; 

3)  Limited  reporting  requirements; 

4)  Small,  high-quality  staffs; 

5)  Close  communication  with  the  user;  and 

6)  Prototyping  and  testing.  (23:50) 

Baumgartner,  Brown,  and  Kelly  conducted  a  study  of  DoD 
programs  that  were  considered  to  be  successful  and 
identified  the  key  factors  that  they  felt  led  to  success: 

1)  Good  people; 

2)  Stable  requirements  and  funding; 

3)  Continuity  of  key  personnel; 

5)  Acquisition  strategy; 

6)  Resources; 

7)  State-of-the-art  technology; 

8)  Use  of  an  integrating  contractor; 

9)  Influence  of  outside  agencies;  and 

10)  DoD  directives  and  regulations.  (1:32) 

A  commonly  held  belief  is  that  technical  problems  have  been 
the  cause  of  cost  and  schedule  growth.  Studies  by  DoD  and 
independent  agencies  show  that,  in  actuality,  the  main  cause 
of  cost  and  schedule  growth  has  been  instability  in 
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establishing  requirements,  planning,  and  budgeting  for 
programs  (18:3). 

Air  Force  acquisition  programs  are,  by  their  nature, 
enormous  undertakings.  Clearly,  no  one  factor  can  be 
singled  out  as  being  responsible  for  program  success  or 
failure;  however,  the  importance  of  communication  with  the 
user  was  a  recurring  theme  in  the  literature  (1;  18;  20;  23; 
26) .  Baumgartner,  Brown,  and  Kelly  surveyed  DoD  program 
managers  for  their  definition  of  program  success. 

Sixty-eight  percent  responded  with  "works  well  when  fielded" 
(1:32).  The  users  will  make  the  ultimate  determination  of 
the  adequacy  of  the  system;  their  involvement  throughout  the 
acquisition  process  is  critical  to  program  success  (20:21). 
This  communication  must  be  a  two  way  process.  The  user  must 
clearly  identify  its  needs  and  the  developer  must  let  the 
user  know  when  requirements  cannot  be  met  so  that  trade¬ 
offs  can  be  made  (26:122).  This  is  an  important 
consideration  even  before  the  program  begins.  The  Packard 
Blue-Ribbon  Commission  found  that,  in  too  many  cases,  the 
user  requirements  for  new  systems  were  overstated  and 
resulted  in  cost  growths  (23:xxiii). 

Commander's  Policies 

The  majority  of  acquisition  within  the  Air  Force  is 
conducted  by  Air  Force  Systems  Command  (AFSC) .  In  1987  the 
AFSC  Commander,  General  Bernard  P.  Randolph,  established  the 
AFSC  550-series  of  regulations  as  a  means  of  documenting  and 
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communicating  his  Command  policies  to  the  members  of  AFSC 
(9:1).  These  regulations  cover  a  variety  of  subjects,  all 
aimed  at  describing  his  philosophy  on  the  way  that  AFSC 
should  conduct  business  in  order  to  be  successful.  The 
second  regulation  in  the  series,  AFSCR  550-2,  identifies 
three  goals  that  AFSC  must  focus  on.  The  importance  of  the 
user  is  clearly  stated  in  the  first  AFSC  goal:  "To  meet  the 
users'  needs  by  keeping  close  to  our  users,  knowledgeable  of 
their  environment,  and  fully  responsive  to  their 
requirements"  (6:1).  This  philosophy  permeates  many  of  the 
regulations  throughout  the  550-series. 

AFSC  must  be  committed  to  focusing  on  the  user.  AFSCR 
550-10  emphasizes  AFSC's  role  of  supporting  the  using 
commands.  It  states  that  program  directors  must  consider 
the  users'  needs  when  making  programmatic  decisions,  brief 
the  AFSC  Commander  on  issues  which  impact  operational 
capability,  and  maintain  close  contact  with  AFOTEC  to  insure 
that  appropriate  operational  test  planning  is  conducted 
(7:1).  AFSCR  550-11  adds  to  this  the  concept  that,  in 
addition  to  simply  meeting  the  users'  needs,  AFSC  should 
strive  to  provide  the  user  with  value.  This  means  that  a 
variety  of  solutions  should  be  proposed  by  AFSC  before  the 
best  solution  is  selected  by  the  user  (10:1;  11:1).  Once 
this  is  accomplished,  the  solution  must  be  pursued  using  a 
cost  effective  acquisition  strategy  in  order  to  provide  the 
most  capability  for  the  money  spent  (8:1). 
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Close  communication  between  the  SPO  and  the  user  is 
facilitated  by  the  existence  of  using  command 
representatives  stationed  at  the  base  where  the  SPO  is 
located.  At  Wright  Patterson  AFB  for  instance,  Aeronautical 
Systems  Division  (ASD)  SPOs  are  supported  by  Military 
Airlift  Command  (MAC) ,  Strategic  Air  Command  (SAC) ,  and 
Tactical  Air  Command  (TAC)  systems  offices  (i.e.  MACSO, 
SACSO,  and  TACSO) .  These  support  offices  advise  the  SPOs  of 
the  using  commands'  interests  and  concerns  relating  to  the 
development  and  future  operation  of  the  weapon  systems.  In 
addition  to  the  support  offices,  many  of  the  SPOs  have  a 
using  command  representative  located  directly  in  the  program 
office. 

Operational  Requirements  Process 

The  weapon  systems  acquisition  process  informally  begins 
when  a  major  command  (MAJCOM) ,  as  the  user,  identifies  an 
operational  deficiency  through  a  mission  analysis.  An 
operational  deficiency  may  result  from  a  myriad  of  reasons 
including: 

1)  changes  in  the  threat  or  national  security  policy, 

2)  technological  advances,  or 

3)  poor  cost  efficiency  or  operational  effectiveness. 

The  identification  of  operational  requirements  is  a  critical 
step.  The  Packard  Blue-Ribbon  Commission  found  that  a 
better  method  of  generating  requirements  was  needed  at  the 
beginning  of  the  acquisition  process  (23:xxiii).  Indeed, 
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the  definition  of  requirements  has  plagued  DoD  for  years. 

In  his  report  to  the  President  in  1970,  Fitzhugh  noted  that 
the  requirements  process  is  critical  at  the  start  of  an 
acquisition  program  (17:68).  The  Air  Force  requirements 
process  was  revised  in  1987  in  response  to  the  observations 
and  recommendations  of  the  1985  Defense  Science  Board  (DSB) 
and  the  1986  Packard  Commission  report.  Additional  cause 
for  change  was  identified  as  a  result  of  the  system 
acquisition  management  inspections  (SAMIs)  that  were 
conducted  between  1979  and  1984.  The  SAMI  inspectors 
documented  55  distinct  cases  where  requirements  were 
improperly  defined  or  communicated  (14:1).  These  findings 
resulted  in  the  direction  of  a  SAMI  of  the  requirements 
process  in  1987  by  The  Inspector  General  (TIG) . 

An  operational  need  may  be  identified  by  either  the  user 
or  HQ  USAF.  Once  an  operational  need  is  validated,  there 
are  many  options  available  to  compensate  for  the  deficiency. 
Among  these  options  are: 

1)  changing  military  doctrine  or  tactics, 

2)  modifying  existing  systems, 

3)  using  modified  commercial  ("off-the-shelf")  equipment, 

4)  developing  a  new  system. 

An  operational  requirement  which  must  be  fulfilled 
through  one  of  the  aforementioned  options  marks  the 
beginning  of  the  long  and  complicated  acquisition  process. 
The  requirement  must  first  be  validated  by  the  originating 


11 


organization  and  is  then  formally  documented  in  a  Statement 
of  Operational  Need  (SON) .  Development  of  the  SON  is 
directed  by  Air  Force  Regulation  (AFR)  57-1,  Operational 
Needs.  Requirements .  and  Concepts  (12:3).  The  SON  describes 
the  need  in  operational  terms  and,  when  approved,  provides 
official  validation  of  the  need. 

The  SON  provides  the  mission  need,  basis  of  the  need, 
assessment  of  current  capabilities,  operational  performance 
requirements,  and  a  proposed  solution  in  a  standardized 
format  (12:29).  Prior  to  being  published,  and  thus 
validated  as  a  legitimate  need,  the  originator  of  the  SON 
must  coordinate  it  with  the  organizations  which  are 
identified  in  Attachment  4  to  AFR  57-1  (12:26).  Approval  of 
a  mission  need  occurs  at  Milestone  0  (MO)  and  allows  the 
effort  to  proceed  into  the  concept  exploration/definition 
phase  (Figure  1) .  Entry  into  this  phase  does  not  guarantee 
that  a  new  weapon  system  will  be  acquired?  it  simply  allows 
alternative  systems  to  be  identified  and  evaluated. 

Three  attachments  must  be  included  with  the  draft  SON:  a 
draft  Program  Decision  Package  (PDP)  which  provides  a 
preliminary  estimate  of  the  funds  required  to  pursue 
development  of  the  weapon  system,  a  threat  assessment,  a  id  a 
Requirements  Correlation  Matrix  (RCM) .  The  RCM  was 
developed  from,  and  is  essentially  the  same  as,  an  Air  Force 
Systems  Command  (AFSC)  and  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC)  initiative  known  as  the  Baseline 
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Figure  1.  Acquisition  Lifecycle 


Correlation  Matrix  (BCM) .  The  RCM  and  BCM  are  management 
tools  which  document  the  user’s  requirements  and  any  changes 
that  occur  to  them  throughout  the  acquisition  process.  One 
striking  difference  between  the  RCM  and  BCM  is  that  the  RCM 
is  developed  by  the  user  (operating  command)  and  the  BCM  is 
developed  by  the  implementing  (developing)  command.  In 
addition,  the  BCM  was  designed  to  document  the  program 
baseline  while  the  RCM  was  designed  to  provide  an  audit 
trail  of  the  requirements  as  the  program  matures. 

Requirements  Correlation  Matrix 

The  RCM  was  developed  to  increase  management  visibility 
of,  and  provide  a  formal  means  for  communication  of,  program 
requirements.  There  are  two  parts  to  the  RCM. 

Part  I  is  a  matrix,  similar  to  a  spreadsheet,  which 
consists  of  four  columns  to  identify  the  top-level 
performance  parameters,  user  requirements,  contractual 
specifications,  and  operational  test  criteria.  The 
requirements  must  be  identified  as  "firm"  or  as  "goals." 
Wherever  possible,  these  requirements  should  be  stated  in 
general  terms  (goals)  to  allow  for  cost  and  performance 
trade-offs  (2:5;  12:9).  The  requirements  are  used  as  a 
basis  for  developing  design  specifications  and  test 
criteria.  This  format  provides  an  easy  means  of 
highlighting  any  disconnects  that  occur  between 
requirements,  specifications,  and  test  criteria. 
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Part  II  provides  the  rationale,  to  include  results  of  any 
trade-off  studies  or  other  factors  considered,  for  any 
changes  in  Part  I  (12:39).  This  section  documents  the 
evolution  of  the  requirements  and  hence  can  be  used  as  an 
audit  trail  for  any  changes  that  occur  throughout  the 
program. 

Baseline  Correlation  Matrix 

The  BCM  was  formalized  in  1987  when  its  governing 
regulation,  AFR  800-46,  Baseline  Correlation  Matrix,  was 
published.  The  BCM  was  designed  to  supplement  the  program 
baseline,  highlight  program  requirements,  and  provide  a 
channel  of  communication  between  the  user,  developer, 
development  tester,  and  operational  tester.  The  BCM,  like 
the  RCM,  contains  two  parts.  Part  I  is  a  five  column 
spreadsheet  which  identifies  the  program  parameters, 
requirements,  specifications,  IOT&E  evaluation  criteria. 

The  fifth  column  is  reserved  for  remarks.  Part  II  contains 
the  rationale  for  the  parameters  identified  in  Part  I  and  a 
description  of  the  methodology  used  to  demonstrate  each 
parameter  (3:5).  The  rationale  sheet  must  identify  the 
original  source  document  for  the  parameters. 

One  of  the  findings  of  the  SAMI  of  the  requirements 
process,  conducted  23  February  to  6  May  1987,  was  that  the 
RCM  and  BCM  were  redundant.  The  SAMI  recommended  that  the 
RCM  and  BCM  be  combined  into  one  document  because  of  the 
similarity  of  information  presented  and  timing  of 
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issuance  (14:10).  The  apparent  duplication  of  information 
in  the  two  documents  resulted  in  the  cancellation  of  the 
requirement  for  a  BCM  in  late  1989. 

Lessons  Learned 

The  Air  Force  Lessons  Learned  Program  was  established  in 
1977  to  provide  a  means  of  documenting  feedback  on  the 
experience  of  current  and  past  acquisition  programs  and  the 
operation  and  support  of  fielded  systems  (4:1).  The  Lessons 
Learned  Data  Bank  is  located  at  Wright  Patterson  AFB,  Ohio, 
and  is  maintained  by  the  Air  Force  Acquisition  Logistics 
Division  (ALD/LSE) .  This  data  bank  maintains  a  collection 
of  approximately  2000  lessons  learned  from  Air  Force,  Army, 
and  Navy  programs  which  are  organized  into  49  subject  areas 
referred  to  as  "impact  areas."  The  impact  areas  are  listed 
in  Appendix  B. 

RCM  Lessons  Learned.  One  lesson  learned  in  the  Data  Bank 
which  discusses  the  importance  of  the  RCM  is  located  in  the 
impact  area  of  Program  Management.  This  lesson,  which 
discusses  the  topic  of  effective  test  planning,  states  that 
understanding  the  operational  requirements  of  the  system  is 
critical  and  that  care  must  be  taken  to  develop  test 
criteria  that  truly  reflect  these  operational  requirements. 
Disconnects  between  the  operational  requirements  and  the 
test  criteria  will  result  in  ineffective  testing  of  the 
system.  The  lesson  concludes  by  stating  that  the  RCM  is  the 
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appropriate  tool  for  documenting  this  process  (15:1).  The 
abstract  for  this  lesson  learned  in  provided  in  Appendix  B. 

The  OT&E  Lessons  Learned  program  is  located  at  Kirtland 
AFB ,  New  Mexico,  and  is  maintained  by  the  AFOTEC  Plans  and 
Policies  Division  (AFOTEC/XPX) .  The  objective  of  the 
program  is  to  ensure  that  future  OT&E  test  efforts  benefit 
from  past  experiences.  The  data  base  contains  management, 
operations  security  (OPSEC) ,  and  technical-oriented  lessons 
learned  for  programs  in  OT&E  or  joint  test  and  evaluation 
( JT&E) .  The  governing  regulation  for  this  program 
recommends  that  lessons  learned  for  AFOTEC-conducted  tests 
be  submitted  at  the  following  times: 

1.  When  the  program  transfers  from  advanced  planning  to 
the  test  manager; 

2.  when  the  program  begins  test  execution;  and 

3.  at  the  end  of  test  execution  and 
reporting/ termination. 

Lessons  learned  for  JT&E  programs  should  be  submitted  at  the 
end  of  the  test  repo  ting  phase  (5:1). 

BCM  Lessons  Learned.  One  lesson  learned  in  the  OT&E 
lessons  Learned  data  base  which  discusses  the  importance  of 
the  BCM  was  submitted  by  the  Automated  Remote  Tracking 
Station  (ARTS)  program.  This  lesson  states  that  rushing  the 
BCM  process  will  result  in  "poorly  designed  and/or 
inappropriate  user  requirements  and  test  criteria."  (13) 

An  additional  issue  that  was  discussed  was  related  to  making 


17 


changes  to  the  BCM  once  it  had  been  signed  by  the  developer. 
Different  interpretations  of  the  user's  requirements,  which 
were  identified  during  IOT&E,  were  difficult  to  resolve 
because  the  changes  to  the  BCM  would  have  to  be  coordinated 
by  the  developers  and  they  did  not  feel  that  they  had 
adequate  time  to  do  so.  The  conclusion  of  this  lesson  was 
that  the  BCM  should  be  written  early  enough  to  preclude 
problems  during  testing  (13).  Another  point  that  stands  out 
is  that  the  criteria  identified  in  the  BCM  must  be  clearly 
written  so  that  there  will  not  be  differing  interpretations. 
The  complete  lesson  learned  is  provided  in  Appendix  B. 

Operational  Test  and  Evaluation  (OT&E) 

OT&E  is  conducted,  by  AFOTEC  or  the  user,  throughout  the 
acquisition  lifecycle  to  assess  the  operational 
effectiveness  and  suitability  of  a  new  weapon  system 
(Figure  2) .  Two  distinct  categories  of  OT&E  exist:  Initial 
OT&E  (IOT&E)  and  Follow-on  OT&E  (FOT&E) .  IOT&E,  the 
earliest  assessment  of  the  system  performance,  occurs  prior 
to  the  full-rate  production  decision  and  is  conducted  on 
production  representative  systems.  FOT&E  occurs  after  the 
full  rate  production  decision  and  is  conducted  during 
production  and  deployment  of  the  system.  The  objective  of 
FOT&E  is  to  verify  the  correction  of  any  deficiencies 
identified  in  IOT&E  and  to  refine  estimates  of  performance. 
The  refinement  of  estimates  is  necessary  due  to  the  inherent 
differences  in  production-representative  and  production 
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systems.  FOT&E  may  continue  throughout  the  life  of  a  system 
to  evaluate  any  future  changes  to  the  system  or  military 
doctrine  and  tactics. 

Effective  use  of  operational  testing  is  essential  to 
improving  the  performance  of  new  systems  (23:xxvi). 
Operational  testing  is  meant  to  be  a  constructive  process 
that  leads  to  the  fielding  of  a  system  that  meets  the  user's 
needs.  Dr.  Kimmel ,  Assistant  Deputy  Director  of  Defense 
Research  and  Engineering  for  Test  and  Evaluation 
(DDDRE (T&E) ) ,  notes  that  operational  testing  has  become  a 
critical  element  of  the  acquisition  process  (21:3).  The 
problem  that  has  arisen  is  that  Congress  tends  to  view 
operational  testing  as  a  pass/fail  exam.  In  this  context,  a 
"failed"  test  may  prompt  Congress  to  cancel  or  reduce 
funding  for  the  program.  The  services  become  wary  of 
conducting  tests  that  are  too  likely  to  fail  (22:57).  In 
other  words,  operational  testing  should  not  be  an  end  unto 
itself  (21:2) . 

The  first  step  that  must  be  taken  to  develop  an  effective 
operational  test  program  is  to  evaluate  the  weapon  system 
and  decide  which  performance  features  are  required  by  the 
user.  These  performance  features  must  be  the  basis  for  the 
operational  evaluation  to  ensure  that  the  system  meets  the 
user's  requirements.  In  other  words,  the  users  must  state 
their  requirements  for  the  system  clearly  so  that  the 
operational  testers  know  what  to  test.  In  the  past,  AFOTEC 
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has  been  faced  with  the  problem  that  a  requirement  has  not 
been  stated  where  they  felt  there  should  be  one  (24) . 
Although  this  approach  filled  the  gap  for  inadequate  or 
inappropriate  requirements,  it  was  not  AFOTEC's 
responsibility  and  it  did  not  fix  the  problem.  AFOTEC  has 
since  taken  the  approach  that  they  will  refuse  to  accept 
inadequate  or  inappropriate  requirements  and  insist  that  the 
user  fulfill  this  responsibility  (24)  .  The  key  performance 
features  for  a  system  that  are  required  by  the  user  and  must 
be  tested  by  AFOTEC  should  be  documented  in  the  RCM/BCM. 
These  requirements  must  then  be  prioritized  to  ensure  that 
the  most  important  features  of  the  system  are  tested  (24) . 
Once  prioritized,  these  requirements  must  be  reviewed  to 
ensure  that  they  reflect  the  operational  capabilities  that 
are  necessary  to  fulfill  the  mission  and  are  not  simply  a 
re-statement  of  the  specifications  (24).  Prioritizing  the 
requirements  for  the  system  and  stating  them  in  operational 
terms  will  help  ensure  that  the  operational  test  program 
provides  a  meaningful  evaluation  of  the  weapon  system  by 
eliminating  unnecessary  and/or  inappropriate  test 
objectives.  A  successful  operational  test  program  will 
determine  whether  or  not  the  system  meets  the  users 
requirements  rather  than  the  appropriateness  of  the  system's 
requirements . 
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Conclusion 


Acquisition  of  new  weapon  systems  within  the  Air  Force  is 
a  complex  process.  Cost  overruns  and  schedule  slips  have 
plagued  the  acquisition  process  for  many  years.  Studies 
have  been  conducted  within  and  outside  DoD  on  military  and 
civilian  programs  to  determine  what  causes  a  program  to 
succeed  or  fail.  One  characteristic  that  is  consistently 
identified  as  essential  to  program  success  is  good 
communication  with  the  user. 

The  developer  is  in  business  to  serve  the  user.  It  is 
essential  that  the  user's  needs  be  carefully  addressed. 
Failure  to  deliver  a  product  which  satisfies  the  user  means 
that  the  developer  has  not  fulfilled  its  duties. 

Establishing  close  communications  with  the  user  is  only  the 
first  step.  The  developer  must  then  listen  to  the  user  and 
be  responsive  to  his  needs.  Trade-offs  must  be  made  when 
the  developer  cannot  deliver  what  the  user  needs. 

The  development  of  user  requirements,  and  the  subsequent 
acquisition  program,  have  an  inherent  flaw.  The  program 
manager  must  "sell"  his  program  to  Congress  to  get  initial 
funding  established.  This  usually  results  in  the 
overstatement  of  the  requirements.  Overstating  requirements 
inevitably  leads  to  program  "failure"  in  the  eyes  of 
Congress  when  the  system  enters  operational  testing  and  does 
not  perform  as  promised.  The  result  is  program 
cancellation,  decreased  funding,  or  schedule  delays  which 
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result  in  cost  overruns.  This  is  a  cyclical  process  which 
often  dooms  the  acquisition  process  to  failure. 

The  RCM  and  BCM  are  the  management  tools  used  to  identify 
the  top-level  performance  parameters  and  the  associated  user 
requirements,  contractual  specifications,  and  operational 
t=>st  criteria.  They  are  used  by  the  developing  command  to 
write  specifications  and  the  operational  tester  to  develop 
the  test  program  and  therefore  are  important  to  the  success 
of  OT&E  and,  in  general,  the  acquisition  program. 

Inadequate  and/or  inappropriate  requirements  in  these 
documents  serve  only  as  an  impediment  to  the  development 
program. 

An  analysis  of  the  selection  of  parameters  found  in 
RCMs/BCMs ,  which  should  reflect  the  user's  requirements,  is 
in  order  to  learn  how  to  improve  the  requirements  process. 

As  was  discovered  in  the  literature,  this  is  not  the 
ultimate  solution  to  the  Air  Force's  program  management 
problems,  but  is  certainly  an  area  which  merits  further 
investigation. 
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Ill .  Methodology 


Overview 

Tnis  research  effort  was  divided  into  two  phases.  The 
first  phase  consisted  of  personal  interviews  and  a  review  of 
literature  on  the  acquisition  process,  with  the  main 
emphasis  placed  on  the  user  and  the  requirements  for  new  or 
improved  weapon  systems.  The  second  phase  constituted  the 
primary  data  collection  for  the  research  effort  and 
consisted  of  an  opinion  questionnaire  distributed  to 
appropriate  subjects.  The  purpose  of  this  chapter  is  to 
limit  the  scope  of  the  research  by  establishing  the 
methodology  that  was  used  in  addressing  the  research 
objectives  during  the  two  phases  of  the  research  effort. 

Phase  One 

The  first  phase  of  the  research  effort  addressed  research 
objective  one.  This  objective  was  to  identify  the  important 
issues  of  the  operational  requirements  process  within  the 
Air  Force  weapon  systems  acquisition  lifecycle.  This  was 
accomplished  through  the  literature  review  and  interviews 
with  AFOTEC  personnel. 

The  literature  reviewed  consisted  of  professional 
military  journals,  DoD  publications.  Air  Force  regulations, 
and  two  lessons  learned  data  bases.  The  professional 
military  journals  were  chosen  because  of  their  focus  on 
current  program  management  issues  relating  to  the 
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acquisition  of  DoD  weapon  systems.  These  publications 
provided  valuable  insight  into  and  analysis  of  the  DoD 
acquisition  process.  The  Air  Force  regulations  served  to 
focus  the  study  by  outlining  the  policies  and  procedures 
that  govern  the  conduct  of  the  requirements  process  within 
the  Air  Force  weapon  system  acquisition  lifecycle.  Finally, 
two  lessons  learned  data  bases  were  searched  to  find 
specific  problems,  and  their  recommended  solutions,  relating 
to  the  requirements  process  that  have  been  encountered  by 
acquisition  personnel.  These  data  bases  included  the  USAF 
Lessons  Learned  Program  at  Wright  Patterson  AFB,  Ohio,  and 
the  OT&E  Lessons  Learned  Program  at  Kirtland  AFB,  New 
Mexico.  One  lesson  learned  in  the  USAF  Lessons  Learned  data 
base  dealt  specifically  with  the  RCM  and  one  lesson  learned 
in  the  OT&E  Lessons  Learned  data  base  dealt  specifically 
with  the  BCM.  These  lessons  learned,  which  are  provided  in 
Appendix  B,  served  to  further  focus  the  study  on  various 
problems  that  have  arisen.  While  the  literature  review 
identified  the  RCM  and  BCM  as  areas  of  potential  study,  the 
subject  required  further  refinement.  This  final  refinement 
was  obtained  by  conducting  personal  interviews  with  key 
AFOTEC  personnel  including  the  AFOTEC  Commander,  Major 
General  Cecil  W.  Powell. 

AFOTEC  was  chosen  because  it  is  the  Air  Force 
organization  tasked  to  evaluate  the  operational  performance 
of  Air  Force  weapon  systems.  The  first  interview  was  of  an 
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informal  and  non-structured  nature.  The  individual 
interviewed,  Mr  Larry  Benson,  is  the  Director  of  the  AFOTEC 
Directorate  of  Research  Services  (AFOTEC/RS) .  This 
directorate  was  chosen  because  two  of  its  functions  provide 
information  directly  relevant  to  this  research.  First,  it 
is  responsible  for  maintaining  the  OT&E  Data  Bank 
(AFOTEC/RSD) ,  which  is  the  USAF  repository  for  OT&E  records. 
Second,  it  is  responsible  for  compiling  the  History  of 
AFOTEC  on  a  yearly  basis.  This  document  serves  as  a 
historical  summary  of  major  events  which  are  pertinent  to 
the  conduct  of  AFOTEC* s  business.  Mr  Benson,  as  head  of 
this  Directorate,  was  a  valuable  source  of  information  on 
issues  relevant  to  the  requirements  process  and  the 
operational  evaluation  of  weapon  systems.  The  purpose  of 
this  interview  was  to  use  his  expertise  on  the  micro  level 
of  the  requirements  process  and  OT&E  to  develop  and  refine 
questions  to  be  addressed  to  Major  General  Cecil  W.  Powell. 
During  the  interview  with  General  Powell,  micro  issues  as 
well  as  some  macro  issues  of  both  the  requirements  process 
and  the  conduct  of  OT&E  were  addressed.  The  purpose  of  this 
second  interview  was  to  better  identify  what  he  felt  were 
problem  areas  of  the  requirements  process  that  affected  the 
conduct  of  OT&E. 

Based  on  the  literature  review  and  these  two  interviews, 
a  questionnaire  was  developed  to  solicit  information  from 
key  personnel  involved  in  the  requirements  process.  The 
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administration  of  this  questionnaire  occurred  during  the 
second  phase  of  the  methodology. 

Phase  Two 

The  second  phase  constituted  the  primary  data  collection 
of  the  research  effort.  This  phase  addressed  research 
objectives  two  and  three  through  the  distribution  of  a 
questionnaire  to  appropriate  personnel. 

Objective  two  was  to  determine  how  and  by  whom  the 
RCM/BCM  parameters  and  the  associated  values,  which  should 
identify  the  user's  requirements,  the  contractual 
specifications,  and  the  IOT&E  criteria  for  the  weapon 
system,  are  selected.  Objective  three  was  to  determine  if 
requirements  personnel  in  the  using  commands,  the  developing 
command,  and  the  operational  test  agency  have  differing 
interpretations  of  both  operational  requirements  and 
specifications.  These  objectives  were  addressed  by 
distributing  a  questionnaire  to  working-level  requirements 
personnel  at  using  commands,  the  developing  command,  and  the 
operational  test  agency. 

Population/Sample .  In  order  to  find  facts  related  to, 
and  opinions  concerning,  the  selection  of  parameters  and 
values  found  in  RCMs  and  BCMs,  it  was  desirable  to  reach  the 
population  which  had  the  greatest  experience  in  this  area. 
For  this  reason,  requirements  personnel  at  using  commands, 
the  developing  command,  and  the  operational  test  agency  were 
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selected.  These  personnel  were  assigned  to  the  following 
organizations: 

1.  HQ  Military  Airlift  Command,  DCS/Requirements 
(HQ  MAC/XR) ,  Scott  AFB,  Illinois. 

2.  HQ  Strategic  Air  Command,  DCS/Requirements 
(HQ  SAC/XR) ,  Offutt  AFB,  Nebraska. 

3.  HQ  Tactical  Air  Command,  DCS/Requirements 
(HQ  TAC/XR) ,  Langley  AFB,  Virginia. 

4.  Aeronautical  Systems  Division,  Directorate  of 
Advanced  Planning  (ASD/XR) ,  Wright  Patterson  AFB,  Ohio. 

5.  HQ  Air  Force  Operational  Test  and  Evaluation  Center, 
Plans  and  Policy  Directorate  (HQ  AFOTEC/XP) ,  Kirtland  AFB, 
New  Mexico. 

A  total  of  95  questionnaires  were  distributed  to  randomly 
selected  individuals  at  these  five  locations.  The  using 
commands  (MAC,  SAC,  and  TAC)  received  60;  ASD  received  20, 
and  AFOTEC  received  15.  The  small  number  of  questionnaires 
sent  to  ASD  and  AFOTEC  was  due  to  the  relatively  low  number 
of  advance  planners  in  these  organizations. 

Questionnaire .  The  purpose  of  the  questionnaire  was  to 
gather  demographics  and  solicit  opinions  about  several 
topics.  The  questionnaire  was  divided  into  two  parts. 

Part  I  consisted  of  a  mix  of  demographics  and  opinion 
questions.  Part  II  consisted  strictly  of  opinion  questions. 

There  were  four  demographic  questions  in  Part  I.  The 
first  question  asked  for  the  current  AFSC,  rank/grade,  and 
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aeronautical  rating.  The  respondent's  aeronautical  rating 
was  solicited  because  the  items  in  Part  II  of  the 
questionnaire  pertained  to  aeronautical  systems  (versus 
space  or  land  systems) .  It  was  used  to  determine  if  those 
with  flying  experience  responded  differently  than  those 
without  flying  experience.  The  second  question  asked  for 
the  respondent's  current  position  title  and  a  brief 
description  of  their  duties.  The  third  question  asked  for 
the  length  of  time  that  the  respondent  had  been  working  in 
their  current  organization.  The  responses  were  ranked  as: 

1.  Less  than  one  year; 

2.  At  least  one  year  but  less  than  two  years; 

3.  At  least  two  years  but  less  than  three  years; 

4.  At  least  three  years  but  less  than  four  years;  or 

5.  More  than  four  years. 

The  fourth  question  asked  for  the  respondent's  previous 
assignment  history.  The  third  and  fourth  questions  were 
primarily  used  to  determine  the  levels  of  current  and  past 
experience . 

The  fifth  and  sixth  questions  asked  the  respondents  to 
give  their  definitions  of  a  requirement  and  a  specification, 
respectively.  The  responses  were  used  to  determine  if  there 
was  a  basic  understanding  of  what  constitutes  a  requirement 
and  a  specification.  They  were  also  used  to  determine  if 
chere  was  a  consensus  opinion  within  the  commands  as  well  as 
between  the  commands. 
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Question  seven  asked  the  respondents  to  identify  the 
basis  used  for  the  definitions  of  a  requirement  and  a 
specification  that  they  provided  in  questions  five  and  six. 
The  purpose  of  this  question  was  to  determine  from  where 
their  understanding  of  the  two  terms  had  come.  Two 
suggestions,  personal  experience  and  professional  continuing 
education  (PCE)  classes,  were  offered  although  the  question 
was  open-ended. 

For  the  AFOTEC  and  ASD  personnel,  question  eight  asked 
the  respondents  to  identify  whether  they  provided  any 
assistance  in  the  selection  of  parameters  and/or  values  to 
the  developers  of  requirements  documents.  This  was  the 
final  question  of  Part  I  for  these  personnel. 

For  the  using  command  personnel,  question  eight  asked  the 
respondents  if  they  personally  were  involved  in  the 
development  of  RCMs.  Question  nine,  the  final  question  of 
Part  I  for  the  using  command  personnel,  asked  the 
respondents  to  identify  if  they  sought  assistance  in  the 
preparation  of  RCMs  from  personnel  outside  of  their 
organization.  Part  I  of  the  questionnaire  is  provided  in 
Appendix  D. 

Part  II  of  the  questionnaire  contained  only  opinion 
questions.  This  part  consisted  of  a  list  of  fifteen 
parameters  and  their  associated  values  as  would  be  found  in 
a  typical  RCM  or  BCM.  These  items  were  selected  from  actual 
RCMs  and/or  BCMs  from  aeronautical  programs  managed  at  ASD. 
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The  parameters  were  taken  from  the  RCMs/BCMs  exactly  as  they 
appeared,  while  the  values  were  changed  slightly  to  protect 
any  sensitive  information  about  the  programs.  The  subjects 
were  asked  to  categorize  each  parameter  and  its  associated 
value  as  a  requirement  (R) ,  a  specification  (S) ,  both  a 
requirement  and  a  specification  (B) ,  or  neither  a 
requirement  nor  a  specification  (N) .  Part  II  of  the 
questionnaire  is  provided  in  Appendix  E. 

Statistical  Tests 

The  data  obtained  from  the  questionnaire  were  analyzed 
using  several  statistical  techniques. 

Frequency  counts  were  obtained  for  the  demographic 
questions  in  Part  I.  These  counts  were  totaled  for  each 
organization  as  well  as  for  the  sample  as  a  whole. 

The  parameter  and  value  categorizations  for  Part  II  of 
the  questionnaire  were  first  totaled  for  the  respondents 
without  any  consideration  of  the  demographics.  The 
parameter  and  value  categorizations  were  then  totaled  based 
on  the  respondent's  aeronautical  rating,  current  experience 
level,  and  organization.  A  statistical  test  for 
independence  was  conducted  on  each  question  for  all  three 
demographic  groups  to  determine  whether  or  not  the  responses 
were  independent  with  respect  to  the  factors. 

The  independence  tests  for  rated  and  non-rated  personnel 
categorizations  considered  the  following  two  hypotheses: 
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H0:  The  categorizations  by  the  rated  personnel  were 

independent  of  those  by  non-rated  personnel. 

Ha:  The  null  hypothesis  is  not  true. 

The  responses  were  totaled  for  the  using  commands,  ASD,  and 

AFOTEC.  The  independence  tests  for  these  response  groupings 

considered  the  following  two  hypotheses: 

H0:  The  categorizations  by  the  personnel  were 

independent  of  their  organization. 

Hg:  The  null  hypothesis  is  not  true. 

Finally,  the  responses  were  totaled  based  on  th_  level  of 

experience  in  the  respondent's  current  organization.  To 

facilitate  the  test,  the  responses  were  grouped  in  three 

groups,  as  opposed  to  five  groups  as  had  been  done 

previously.  These  groups  were 

1)  less  than  two  years, 

2)  two  years  but  less  than  four,  and 

3)  four  years  or  more. 

The  independence  tests  for  these  response  groupings 

considered  the  following  two  hypotheses: 

H0:  The  categorizations  by  the  personnel  were 

independent  of  experience  level. 

Ha:  The  null  hypothesis  is  not  true. 

A  two-way  contingency  table  was  constructed  for  each 

question  with  i  factors  and  j  categories.  The  test 

statistic  was  a  Chi-squared  value  (16:587): 

sum,,  sumj  (nu  -  4U) 
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where 


n 


=  sample  size 


n(j  =  the  number  of  responses  in  sample  i  that 
fell  in  category  j 

e( j  =  estimated  expected  count  in  cell  (i,j) 

=  ni*nj/n 

ns  =  the  total  for  row  i 
nj  =  the  total  for  column  j 
The  decision  rule  was  (16:636): 

reject  H0  if  X2  >= 

The  values  of  X2  and  X2alpha>(I.1)*(J.1)  were  compared  for  each 
test  and  the  decision  rule  was  applied.  If  the  null 
hypothesis  is  rejected,  it  can  be  concluded  that  the  two 
groups  are  not  independent  and  that  their  responses  differed 
significantly.  If  the  null  hypothesis  is  not  rejected,  it 
can  be  concluded  that  the  categorizations  by  the  different 
response  groupings  did  not  differ  significantly. 
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IV.  Findings  and  Analysis 


Overview 

This  chapter  presents  the  findings  and  analysis  of  the 
data  collected  through  the  questionnaire  which  was 
distributed  to  personnel  at  AFOTEC,  ASD,  and  the  using 
commands  (MAC,  SAC,  and  TAC) .  The  data  from  the 
questionnaire  was  analyzed  using  the  methodology  described 
in  Chapter  III.  The  questionnaire  response  data  is 
presented  first,  followed  by  the  applicable  questionnaire 
findings  and  analysis. 

Questionnaire  Response  Data 

Table  1  displays  the  participation  results  for  the 
questionnaire.  All  of  the  questionnaires  returned  were 
received  by  the  cutoff  date,  31  July  1990. 

TABLE  1 

Participation  Results 


Number 

Percentage 

Questionnaires 

Distributed 

95 

— 

Questionnaires 

Returned 

53 

55.8 

AFOTEC 

7 

46.7 

ASD 

14 

70.0 

Using  Commands 

32 

53.3 
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Questionnaire  Findings  and  Analysis 


Demographic  Questions.  The  first  series  of  questions 
asked  for  some  demographic  information  about  the 
respondents . 

Table  2  shows  the  flying  rating  of  the  respondents.  The 
number  of  rated  and  non-rated  respondents  was  fairly  close; 
slightly  more  respondents  were  non-rated.  Table  3  shows  the 
ranks/grades  of  the  respondents.  One  of  the  53  respondents 
did  not  provide  an  aeronautical  rating  and  two  did  not 
identify  their  rank/grade. 


TABLE  2 

Aeronautical  Rating  (%) 


Using 

Rating 

Commands 

ASP 

AFOTEC 

TOTAL 

Rated 

45.2 

21.4 

71.4 

42.3 

Non-rated 

54.8 

78.6 

28.6 

57.7 

TABLE  3 

Ranks/Grades  (%) 

Using 

Rank/Grade 

Commands 

ASP 

AFOTEC 

TOTAL 

Lt  Col/GM-14 

23.3 

50.0 

28.6 

31.4 

Maj/GS-13 

26.7 

28.6 

42.8 

29.4 

Capt/GS-12 

20.0 

14.3 

28.6 

19.6 

Lt 

6.7 

7.1 

0.0 

5.9 

NCO 

23.3 

0.0 

0.0 

13.7 
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Table  4  shows  the  length  of  time  that  the  respondents 
have  been  working  in  their  requirements  organization. 
Exactly  half  of  the  respondents  reported  that  they  had  less 
than  two  years  experience  in  their  current  organization. 


TABLE  4 

Current  Organization  Experience  Level  (%) 


Level  (years) 

Using 

Commands 

ASD 

AFOTEC 

Total 

L  <  1 

29.0 

14 . 3 

28.6 

25.0 

1  <=  L  <  2 

35 . 5 

7 . 1 

14.2 

25.0 

2  <=  L  <  3 

16.1 

21.5 

28.6 

19.2 

3  <=  L  <  4 

19.4 

7 . 1 

0.0 

13.5 

L  >  4 

0.0 

50.0 

28.6 

17.3 

Slightly  less  than  half  (45.2%)  of  the  personnel  in  the 
using  command  organizations  identified  that  they  had  an 
aeronautical  rating.  This  might  seem  unusually  low  given 
the  fact  that  the  commands  are  operationally  oriented; 
however,  this  might  be  explained  by  the  fact  that  these 
organizations  are  staff  functions.  The  personnel  in  these 
organizations  also  tended  to  be  higher  ranking.  Exactly 
half  (50%)  of  the  personnel  were  major/GS-13  or  lieutenant 
colonel/GS-14 .  A  rank  of  major  generally  requires  at  least 
10  years  of  Air  Force  experience  to  obtain;  the  grade  of  GS- 
13  might  require  slightly  less.  The  respondents  that 
identified  themselves  as  non-commissioned  officers  (NCOs) 
were  all  master  sergeant  or  above  and  accounted  for  another 
23.3  percent.  As  a  minimum,  these  ranks  also  require  at 
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least  10  years  of  Air  Force  experience  to  obtain.  On  the 
other  hand,  none  of  these  personnel  had  more  than  four  years 
experience  in  the  their  current  organization  and  over  naif 
(64.5%)  of  the  respondents  indicated  that  they  had  less  than 
two  years  of  experience  in  their  current  organization. 

The  demographics  for  the  ASD  and  AFOTEC  indicate  a 
different  profile.  The  number  of  higher  ranking  personnel 
was  greater  than  for  the  using  commands?  ASD  reported  78.6% 
and  AFOTEC  reported  71.4%  as  being  major/GS-13  and  above. 

The  level  of  experience  for  the  ASD  and  AFOTEC  respondents 
was  also  distributed  differently  than  was  the  using 
commands'.  While  over  half  of  the  using  command  respondents 
indicated  having  less  than  two  years  of  experience  in  their 
current  organization,  the  reverse  was  shown  for  ASD  and 
AFOTEC.  ASD  reported  78.6%  and  AFOTEC  reported  57.2%  of 
their  personnel  having  two  or  more  years  of  experience  in 
their  current  organization.  In  fact,  half  of  the  ASD 
respondents  and  a  quarter  of  the  AFOTEC  respondents  had  more 
than  four  years  experience  in  their  current  organization. 
This  was  due  to  the  larger  number  of  civilian  personnel  in 
these  organizations. 

Opinion  Questions.  The  remaining  questions  of  Part  I 
asked  respondents  to  provide  their  opinion  on  several  items, 
including  the  definitions  of  a  requirement  and  a 
specification,  and  where  they  acquired  those  definitions. 
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Additionally,  it  addressed  the  level  of  assistance  provided 
during  RCM/BCM  preparation. 

One  question  asked  respondents  for  their  definition  of  a 
requirement.  The  responses  for  this  question  are  provided 
in  Appendix  F.  The  definitions  of  a  requirement  provided  by 
the  respondents  varied  greatly.  The  definitions  tended  to 
describe  a  requirement  on  three  different  levels.  The  first 
level  was  that  of  a  requirement  as  an  operational 
characteristic  or  capability  of  a  system.  The  second  level 
was  that  of  a  requirement  as  a  solution,  in  the  form  of  a 
system,  to  an  operational  need  or  deficiency.  Finally,  the 
third  and  the  highest  level  was  that  of  a  requirement  as  a 
user's  operational  need  or  a  mission  capability.  The 
responses  from  AFOTEC  were  almost  wholly  oriented  towards 
the  first  level  while  those  from  ASD  and  the  using  commands 
were  a  mix  of  all  three  levels. 

A  second  question  asked  respondents  for  their  definition 
of  a  specification.  The  responses  for  this  question  are 
provided  in  Appendix  G.  Specifications  can  generally  be 
characterized  as  either  functional,  design,  or  performance 
in  nature,  or  a  combination  of  the  three.  Functional 
specifications  describe  the  essential  physical 
characteristics  and  functions  necessary  to  meet  minimum 
needs.  Design  specifications  describe  in  detail  the 
materials,  dimensions,  and  manufacturing  process  for  an 
item.  Performance  specifications  describe  requirements  in 
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terms  of  ranges  of  acceptable  characteristics  or  minimum 
acceptable  standards.  While  some  of  the  definitions  of  a 
specification  were  focused  solely  on  the  performance  aspect, 
they  were  few  in  number.  Most  of  the  definitions 
encompassed  all  three  aspects. 

Table  5  shows  the  basis  that  was  used  for  the  definitions 
of  a  requirement  and  a  specification  that  were  provided  by 
the  respondents.  The  table  shows  the  totals  for  each 
location  as  well  as  the  overall  total. 


TABLE  5 

Basis  of  Definitions  (%) 


Basis 

Using 

Commands 

ASD 

AFOTEC 

TOTAL 

Personal 

67 . 4 

59.1 

87.5 

67.1 

Experience 
PCE  Classes 

23.3 

18.2 

0.0 

19.2 

Other 

9.3 

22.7 

12.5 

13.7 

The  questionnaire  provided  personal  experience  and 
professional  continuing  education  (PCE)  classes  as  examples 
to  provide  clarification  of  what  was  being  asked.  Although 
the  question  was  open-ended  beyond  these  suggestions,  the 
respondents  chose  them  most  often.  Also,  respondents  were 
not  limited  to  providing  only  one  response  for  this 
question.  For  this  question,  there  were  43  responses  from 
the  using  commands,  22  responses  from  ASD,  and  eight 
responses  from  AFOTEC  for  a  total  of  73  responses.  The 
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third  category,  "other,"  was  included  to  account  for  things 
such  as  regulations  and  a  dictionary. 

The  responses  for  all  three  groups  was  heavily  weighted 
towards  personal  experience  as  the  basis  (67.1%). 
Considerably  less  (19.2%)  indicated  that  PCE  classes 
(e.g.  AFIT  SYS  100-400  and  DSMC)  were  an  influence.  The 
remaining  responses  (13.7%)  were  grouped  into  the  category 
"other."  This  category  included  things  such  as  AFR  57-1, 

MIL  STD  490A,  and  a  dictionary. 

Table  6  shows  the  results  of  the  question  concerning 
assistance  for  the  seven  AFOTEC  and  fourteen  ASD 
respondents.  This  question  asked  respondents  if  they 
provided  assistance  in  the  selection  of  parameters  and/or 
values  to  the  developers  of  ROMs  and  BCMs.  AFOTEC  indicated 
slightly  more  involvement  than  did  ASD,  although  both  were 
close  to  50  percent. 


TABLE  6 

Assistance  in  Selection  of  Parameters/Values  (%) 


AFOTEC 

ASD 

Total 

Provide  assistance 

57.1 

54 . 5 

55.6 

Do  not  provide  assistance 

42.9 

45.5 

44.4 

Table  7  shows  the  results  of  the  question  concerning 
involvement  for  the  31  using  command  (MAC,  SAC,  and  TAC) 
respondents.  This  question  asked  respondents  if  they  were 
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currently  involved  in  the  development  of  user  requirements 
for  RCMs . 


TABLE  7 


Development 

of  User 

Requirements 

for  RCMs 

(%) 

MAC 

SAC 

TAC 

Total 

Involved 

77.8 

78.6 

75.0 

77.4 

Not  involved 

22.2 

21.4 

25.0 

12.6 

Slightly  more  than  half  (55.6%)  of  the  ASD  and  AFOTEC 
respondents  indicated  that  they  provided  assistance  in  the 
selection  of  parameters  and/or  values  to  the  developers  of 
ROMs  and  BCMs.  Conversely,  the  using  commands  indicated 
that  a  fairly  high  number  of  personnel  (77.4%)  were 
"currently"  involved  in  the  development  of  requirements  for 
RCMs.  This  figure  would  have  been  slightly  higher  had  the 
word  "currently"  been  omitted;  some  of  the  responses  counted 
as  "not  involved"  were  stated  as  "not  currently."  This 
figure  indicates  that  a  large  number  of  the  personnel  had 
some  exposure  to  the  RCM  in  the  conduct  of  their  work, 
despite  the  fact  that  a  relatively  large  number  of  the 
personnel  had  less  than  two  years  of  experience  in  their 
current  organization. 

The  final  question  for  the  using  command  personnel  asked 
them  to  identify  from  whom  they  sought  assistance  when 
developing  requirements  documents.  The  responses  generally 
included  various  using  command  staff  offices,  product 
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division  engineering  offices,  operational  testers,  and 
operating  units.  On  the  other  hand,  comments  about  the 
coordination  process  ranged  from  "poor  response"  to  "more 
help  than  I  need."  The  responses  for  this  question  are 
provided  in  Appendix  H. 

Categorization  Questions.  The  second  part  of  the 
questionnaire  contained  only  opinion  questions  and  consisted 
of  a  list  of  fifteen  parameters  and  their  associated  values 
as  would  be  found  in  a  typical  RCM  or  BCM.  Respondents  were 
asked  to  categorize  each  parameter  and  its  associated  value 
as  a  requirement  (R) ,  specification  (S) ,  both  a  requirement 
and  a  specification  (B) ,  or  neither  a  requirement  nor  a 
specification  (N) .  These  items  were  selected  from  actual 
RCMs  and  BCMs  for  aeronautical  programs  managed  at  ASD.  The 
parameters  were  taken  from  the  RCMs/BCMs  exactly  as  they 
appeared,  while  the  values  were  changed  slightly  to  protect 
any  sensitive  information  about  the  programs.  Figures  3,  4, 
5,  and  6  show  the  relative  frequency  histograms  of  the 
categorizations  of  the  parameters  and  values  given  for  Part 
II  of  the  questionnaire  for  all  of  the  respondents. 

Appendix  I  shows  the  responses  in  tables  of  percentages. 

The  relative  frequency  histograms  in  figures  3,  4  ,  5,  and  6 
were  prepared  using  the  Ouattro  Pro  spreadsheet  program 
(25)  . 

Appendix  J  shows  the  categorizations  of  the  parameters 
and  values  in  Part  II  of  the  questionnaire  for  the 
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respondents  grouped  by  aeronautical  rating.  The  percentages 
in  these  tables  were  calculated  based  on  the  aeronautical 
rating  alone.  Similarly,  the  responses  are  shown  as 
percentages  for  each  organization  in  Appendix  K  and  for  the 
three  experience  level  groups  in  Appendix  L. 


43 


PERCENT  |  w|  PERCENT 


Parameter  Categorization  Histogram  for 
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Z53  REQ.  (R)  SPEC.  (S'  ■■  BOTH  (B)  □  NEITHER  (N) 

Value  Categorization  Histogram  for 
Questions  1  to  8 


Figure  6.  Value  Categorization  Histogram  for 
Questions  9  to  15 
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The  results  for  the  independence  test  for  categorizations 
grouped  by  aeronautical  rating  are  presented  in  Table  8.  At 
the  level  of  significance,  alpha  =  .05,  the  critical  value 
of  the  test  statistic  used  in  the  decision  rule  was  7.779 
and  was  obtained  from  a  table  of  Chi-squared  critical  values 
(16:636).  The  test  for  each  question  resulted  in  accepting 
the  null  hypothesis  and  indicates  that  there  is  not  a 
significant  difference  in  the  categorizations  by  the  rated 
and  non-rated  respondents. 


TABLE  8 

Independence  Test  Results  of 
Categorizations  by  Aeronautical  Rating 


Ouestion 

X2.  Parameters 

X2.  Values 

1 

.244 

2 . 370 

2 

.982 

2.895 

3 

1.434 

1.509 

4 

3.622 

.731 

5 

1.934 

3.212 

6 

2.139 

2.774 

7 

2.534 

.275 

8 

1.781 

2 .268 

9 

1.905 

.481 

10 

2 .449 

.  135 

11 

5.090 

.835 

12 

1.148 

3 . 298 

13 

2 .240 

2.534 

14 

3.477 

2.738 

15 

7.107 

5.120 

The  results  for  the  independence  test  for  categorizations 
grouped  by  organization  type  are  presented  in  Table  9.  At 
the  level  of  significance,  alpha  =  .05,  the  critical  value 
of  the  test  statistic  used  in  the  decision  rule  was  12.592 
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and  was  obtained  from  a  table  of  Chi-squared  critical  values 
(16:636).  The  test  for  each  question  resulted  in  rejecting 
the  null  hypothesis  for  the  parameter  categorization  of 
question  3  and  the  value  categorization  of  question  11.  The 
remaining  28  tests  resulted  in  accepting  the  null 
hypothesis,  which  indicates  that  there  was  not  a  significant 
difference  in  the  categorizations  by  the  respondents  when 
grouped  by  organization. 


TABLE  9 

Independence  Test  Results  of 
Categorizations  by  Organization 


Question  X2,  Parameters  X2,  Values 


1 

6.378 

3 . 530 

2 

6.377 

1.227 

3 

20.790 

6.368 

4 

6.512 

8.638 

5 

3.859 

7.144 

6 

9.202 

5.308 

7 

7.544 

9.075 

8 

10.741 

4 .210 

9 

11.642 

8.655 

10 

3 . 974 

7.399 

11 

2 . 858 

14.147 

12 

8.813 

3.858 

13 

5.142 

2.539 

14 

9.072 

4.667 

15 

5.604 

3.252 

The  results  for  the  independence  test  for  categorizations 
grouped  by  their  experience  level  in  their  current 
organization  are  presented  in  Table  10.  At  the  level  of 
significance,  alpha  =  .05,  the  critical  value  of  the  test 
statistic  used  in  the  decision  rule  was  12.592  and  was 
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obtained  from  a  table  of  Chi-squared  critical  values 
(16:636).  The  test  for  each  question  resulted  in  rejecting 
the  null  hypothesis  for  the  value  categorizations  of 
questions  2,  11,  and  15.  The  tests  for  these  three 
questions  resulted  in  rejecting  the  null  hypothesis  which 
indicates  that  there  is  a  significant  difference  in  the 
categorizations  when  grouped  by  the  respondents'  experience 
level  in  their  current  organization.  The  remaining  27  tests 
resulted  in  accepting  the  null  hypothesis  which  indicates 
that  there  is  not  a  significant  difference  in  the 
categorizations  when  grouped  by  the  respondents'  experience 
level  in  their  current  organization. 


TABLE  10 

Independence  Test  Results  of 
Categorizations  by  Experience  Level 


Question  X2,  Parameters  X2.  Values 


1 

7.050 

12.369 

2 

2.074 

18.530 

3 

2.939 

11.016 

4 

4.641 

10.693 

5 

2 .869 

6.533 

6 

1.644 

8.340 

7 

1.856 

8.227 

8 

1.428 

9.669 

9 

3.940 

7.305 

10 

4 . 051 

7.376 

11 

1.968 

16.717 

12 

3.515 

5.547 

13 

3.249 

8.404 

14 

5.079 

11.662 

15 

1.639 

13.184 

48 


The  results  of  Part  II  of  the  questionnaire  generally 
indicated  that  there  was  no  consensus  as  to  how  a  parameter 
or  a  value  should  be  categorized.  The  parameter 
categorizations  were  consistently  ranked  as  neither  a 
requirement  nor  a  specification  by  26  to  45  percent  of  the 
respondents.  Although  all  of  the  parameters  are  "user 
requirements, "  each  of  the  15  parameters  was  categorized  as 
a  requirement  by  an  average  of  only  35%  of  the  responses. 

In  general,  the  respondents  (regardless  of  aeronautical 
rating,  organization,  and  experience  level)  believed  the 
parameters  were  neither  requirements  nor  specifications. 

The  opinion  on  the  categorizations  of  the  values  also 
indicated  a  general  lack  of  consensus,  although  there  was  a 
greater  tendency  for  the  values  to  be  termed  a  requirement, 
a  specification,  or  both  than  to  be  termed  neither.  Many  of 
the  respondents  who  chose  "both"  indicated  that  the  value 
was  too  constrained  and  therefore  could  be  either  a 
requir  ment  or  a  specification. 

Summary 

The  typical  using  command  requirements  personnel  had  at 
least  ten  years  of  Air  Force  experience,  but  less  than  two 
years  of  experience  in  their  current  organization.  More 
than  half  of  these  personnel  did  not  have  an  aeronautical 
rating,  although  they  develop  the  requirements  for 
aeronautical  systems.  The  typical  ASD  advance  planners  also 
had  at  least  ten  years  of  Air  Force  experience  and  half  had 


49 


more  than  four  years  of  experience  in  their  current 
organization.  Most  of  the  ASD  respondents  did  not  have  an 
aeronautical  rating.  The  typical  AFOTEC  advance  planners 
had  at  least  ten  years  of  Air  Force  experience,  but  less 
than  three  years  of  experience  in  their  current 
organization.  Most  of  the  AFOTEC  personnel  had  an 
aeronautical  rating. 

The  lack  of  a  clear  understanding  of  parameters  and 
values  as  requirements  or  specifications  is  evident  in  the 
second  part  of  the  study.  In  general,  the  respondents, 
regardless  of  aeronautical  rating,  organization,  and 
experience  level,  were  inconsistent  with  regard  to  the 
categorizations . 

The  implications  of  these  findings  and  the  answers  to  the 
original  research  objectives  are  provided  in  the  next 
chapter. 
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V.  Conclusions  and  Recommendations 


Overview 

The  general  question  behind  this  thesis  was  "How  can  the 
acquisition  process  be  improved?"  The  more  specific 
question  of  this  research  was  "What  improvements  can  be  made 
to  better  define  and  document  the  operational  performance 
requirements  associated  with  the  acquisition  of  weapon 
systems?"  To  address  this  question,  three  research 
objectives  were  established.  This  chapter  presents  the 
conclusions  to  these  research  questions  by  addressing  each 
of  the  three  research  objectives.  Additionally, 
implications  for  managers  are  made  based  on  the  results  of 
the  research  effort.  Finally,  recommendations  for  further 
study  in  this  area  are  presented. 

Research  Objective  Or.e 

Identify  the  important  issues  of  the  operational 
requirements  definition  process  within  the  Air  Force  weapon 
systems  acquisition  lifecycle. 

The  using  commands  are  required  to  conduct  mission 

analyses  on  a  continuing  basis  to  make  a  determination  of 

the  capability  of  weapon  systems  to  meet  the  projected 

threat.  An  operational  deficiency  occurs  when  the  current 

weapon  systems  cannot  meet  this  threat.  This  operational 

deficiency,  or  conversely  operational  need,  must  be 

documented  in  the  Statement  of  Operational  Need  (SON) .  The 
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key  feature  of  the  SON  is  that  it  states  the  operational 
performance  capabilities  that  are  required  to  meet  the 
threat.  The  Requirements  Correlation  Matrix  (RCM) ,  which  is 
a  required  attachment  to  the  SON,  provides  a  means  for 
documenting  the  required  operational  performance  features 
(parameters)  as  the  "user's  requirements."  Like  the  SON, 
the  RCM  must  state  the  user's  requirements  in  operational 
terms.  These  requirements  are  then  translated  into  system 
specif ications  and  IOT&E  evaluation  criteria.  The  RCM 
identifies  these  three  items  in  one  document  and  thus 
provides  a  means  for  highlighting  any  disconnects  that  may 
occur  between  them.  As  was  documented  in  the  literature 
review,  the  interview  with  Major  General  Powell,  and  the 
lessons  learned,  the  selection  of  user  requirements  plays  a 
critical  role  in  the  success  of  the  acquisition  program. 

The  RCM  pro\ ides  an  excellent  tool  for  focusing  on  the  true 
requirements  of  the  weapon  system  being  procured. 

Research  Objective  Two 

Determine  who  selects  the  RCM/BCM  parameters  and  the 
associated  values  which  should  identify  the  user's 
requirements,  the  contractual  specifications,  and  the  IOT&E 
criteria  for  the  weapon  system. 

As  directed  by  AFR  57-1,  the  RCM  is  written  by  the  user 
and  is  included  as  an  attachment  to  the  SON.  The  using 
commands  examined  in  this  research  effort  (MAC,  SAC,  and 
TAC)  each  have  an  organization  which  is  responsible  for  the 
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identification  of  the  operational  requirements  for 
new/modified  weapon  systems.  They  are: 

1.  HQ  MAC/XR,  Scott  AFB,  Illinois; 

2.  HQ  SAC/XR,  Offutt  AFB,  Nebraska;  and 

3.  HQ  TAC/XR,  Langley  AFB,  Virginia. 

The  personnel  from  these  organizations  are  responsible  for 
the  selection  of  parameters  and  the  user's  requirements  for 
them.  The  personnel  from  the  developing  command  (ASD)  are 
responsible  for  translating  the  user's  requirements  into 
specifications  to  serve  as  guidelines  for  the  manufacture  of 
the  solution  by  the  contractor.  Finally,  the  personnel  from 
the  operational  test  agency  (AFOTEC)  are  responsible  for 
developing  IOT&E  criteria  which  are  used  to  determine  if  the 
solution  meets  the  user's  needs. 

The  personnel  at  these  organizations  tended  to  be  higher 
ranking  officers  or  non-commissioned  officers.  An 
implication  of  this  is  that  these  personnel  have  at  least  10 
years  of  Air  Force  experience.  On  the  other  hand,  50%  of 
the  respondents  had  less  than  two  years  of  experience  in 
their  current  organization.  This  disparity  was  especially 
noticeable  when  looking  solely  at  the  using  command 
personnel,  where  64.5%  of  the  respondents  had  less  than  two 
years  of  experience  in  their  current  organization.  The 
requirements  personnel  appear  to  be  highly  experienced  in 
the  broad  sense,  while  only  being  moderately  experienced  in 
their  current  positions. 
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Additionally,  slightly  less  than  half  of  the  respondents 
indicated  that  they  had  an  aeronautical  rating.  One  might 
expect  that  the  personnel  responsible  for  identifying 
operational  requirements  would  have  experience  as  an 
operator;  however,  that  was  not  the  case.  Moreover,  it  may 
not  need  to  be  the  case.  This  research  found  that  rated 
officers  did  not  categorize  the  parameters  and  values 
significantly  different  than  non-rated  personnel. 

Research  Objective  Three 

Determine  if  requirements  personnel  at  using  commands, 
the  developing  command,  and  the  operational  test  agency  have 
differing  interpretations  of  both  operational  requirements 
and  specifications. 

The  requirements  personnel  had  different  interpretations 
of  an  operational  requirement  and  a  specification, 
regardless  of  their  organizational  affiliation.  There  was 
no  consensus  in  the  categorizations  of  the  parameters  or  the 
values  in  the  second  part  of  the  questionnaire.  For 
example,  the  parameters  were  more  often  categorized  as 
neither  a  requirement  nor  a  specification,  than  as  either  a 
requirement  or  a  specification  or  both.  All  of  the 
parameters  should  have  been  identified  as  requirements 
because,  by  definition,  they  are  "user  requirements." 
However,  in  general  they  were  categorized  as  a  requirement 
35%  of  the  time,  and  in  no  cai-e  was  it  by  more  than  54%  of 
the  respondents. 
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The  definitions  of  a  requirement  provided  by  the 
respondents  varied  greatly  but  could  be  classified  into  a 
hierarchy  of  three  distinct  levels: 

1.  an  operational  characteristic  of  a  system; 

2.  a  system  that  will  meet  an  operational  deficiency; 

3.  a  mission  capability. 

AFOTEC  respondents  identified  a  requirement  primarily  on  the 
most  specific  level,  as  an  operational  characteristic  of  a 
system.  The  ASD  and  using  command  responses  varied  among 
the  three  levels.  In  the  context  of  systems  acquisition, 
none  of  these  definitions  are  wrong;  however,  in  the  context 
of  the  RCM/BCM,  the  first  level  seems  to  be  the  most 
appropriate. 

The  definitions  of  a  specification  were  much  more 
consistent.  Most  of  the  definitions  tended  to  include 
design,  functional,  and  performance  characteristics  as 
essential  elements  of  a  specification. 

The  selection  of  parameters  has  a  tremendous  impact  on 
the  rest  of  the  matrix.  The  values  (either  requirements, 
specifications,  or  IOT&E  criteria)  in  the  RCM/BCM  are  only 
criteria  associated  with  a  parameter.  Because  of  the 
different  interpretations  of  a  requirement  and  a 
specification,  and  the  lack  of  clear  and  consistent 
definitions,  the  validity  of  the  parameters  and  the  utility 
of  the  values  in  the  RCM/BCM  need  to  be  questioned. 
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The  specific  question  of  this  research  was  "What 
improvements  can  be  made  to  better  define  and  document  the 
operational  performance  requirements  associated  with  the 
acquisition  of  weapon  systems?”  The  discussion  of  the  three 
research  objectives  indicate  that  there  are  problems  in  the 
process  and  that  improvements  can  be  made.  Based  on  the 
findings  of  this  research,  three  implications  for  managers 
have  been  identified. 

Implications  for  Managers 

1.  The  organizations  responsible  for  the  development  of 
requirements  for  RCMs  and  BCMs  must  have  personnel  who 
understand  the  implications  of  the  selection  of  parameters 
for  the  weapon  system.  The  parameters  should  encompass  the 
essential  characteristics  of  the  system  and  must  accurately 
reflect  the  user's  requirements. 

2.  More  acquisition  training  is  required  for  the 
requirements  personnel,  especially  for  those  in  the  using 
commands.  The  majority  of  the  respondents  (67.1%) 
identified  that  they  formed  their  definitions  of  a 
requirement  and  a  specification  primarily  from  personal 
experience.  PCE  classes,  from  which  Air  Force  personnel 
receive  most  of  their  formal  Air  Force  education  in 
acquisition,  was  cited  by  approximately  19  percent  of  the 
respondents  as  the  source.  The  final  category,  "other,” 
which  included  regulations  and  directives,  was  identified  by 
approximately  14  percent  of  the  respondents.  The  low 
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response  for  PCE  classes  had  two  possible  explanations.  The 
first  explanation  might  be  that  PCE  classes  were  not 
available  to  the  personnel  and  therefore  they  learned  their 
duties  through  "on-the-job  training."  This  is  a  distinct 
possibility  given  the  relatively  limited  availability  of  the 
classes.  The  second  explanation  could  be  that  PCE  classes 
were  available  but  an  in-depth  discussion  of  RCMs, 
requirements,  and  specifications  was  not  included.  A 
for  ial  course  should  be  developed  which  includes  material  on 
the  requirements  process  beginning  with  the  identification 
of  an  operational  need  to  the  fielding  of  the  system.  The 
course  should  focus  on  the  importance  of  the  user's  role  in 
the  systems  acquisition  lifecycle. 

3 .  A  formal  review  of  the  requirements  at  the  general 
officer  level  should  be  conducted  for  major  systems. 

General  Powell  believes  that  the  failure  of  senior  officers 
to  prioritize  requirements  has  been  one  of  the  greatest  past 
weaknesses  of  the  requirements  process.  This  review  is 
essential  to  ensuring  that  the  system  requirements  are  truly 
mission  essential  and  do  not  include  "nice-to-have"  items. 

Further  Study 

This  research  effort  has  addressed  only  a  small  portion 
of  the  requirements  process,  specifically  the  RCM  and  BCM. 
The  following  suggestions  are  presented  in  the  hope  that 
future  research  will  be  accomplished  in  this  area. 
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1.  The  acquisition  process  was  undergoing  significant 
revisions  while  this  research  was  being  conducted.  The 
governing  directive  for  the  acquisition  process, 

DODD  5000.1,  was  undergoing  a  complete  rewrite,  the  purpose 
of  which  was  to  consolidate  the  vast  number  of  acquisition 
regulations  that  had  existed  and  incorporate  changes  to  the 
acquisition  process.  An  examination  of  these  changes  to  the 
acquisition  process  that  result  from  this  revision  should  be 
conducted  to  analyze  the  impacts  to  the  requirements 
process . 

2.  A  study  involving  a  larger  sample  from  AFOTEC  and  ASD 
would  help  guard  against  a  bias  towards  the  using  commands. 
Individuals  from  the  AFOTEC  Test  and  Evaluation  directorate 
and  the  ASD  SPOs  would  be  appropriate  inclusions. 

3.  Part  II  of  the  questionnaire  consisted  of  a  list  of 
fifteen  parameters  and  one  of  their  associated  values 
(requirement  or  specification) .  These  parameters  and  values 
were  randomly  selected  from  actual  RCMs  and/or  BCMs;  no 
criteria  were  used  to  select  the  items.  As  a  result, 
inappropriate  examples  may  have  been  included  in  the 
questionnaire.  A  questionnaire,  similar  to  Part  II,  using 
carefully  selected  parameters  and  values  might  provide 
clearer  insight  into  the  categorizations. 

Collectively,  the  knowledge  gained  from  these  efforts 
could  go  a  long  way  to  improve  the  identification  and 
documentation  of  requirements  in  the  RCM/BCM.  Improvements 
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in  the  RCM/BCM  process  will  have  a  direct  effect  on  the 
military  utility  of  weapon  systems  acquired  by  the  DOD. 
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Appendix  A:  Interview.  Major  General  Cecil  W.  Powell. 
AFOTEC/CC.  18  January  1990 


1.  What  have  been  the  greatest  past  weaknesses  of  the 
requirements  process? 

Basically,  there  have  been  two  weaknesses  in  the  past 
that  have  driven  the  results  of  our  requirements  process. 

The  first,  although  it  recently  is  changing,  has  been 
the  failure  of  senior  officers  to  take  appropriate  action  on 
requirements  generated  by  their  subordinate  levels  of 
command.  That  is,  they  failed  to  review  those  requirements 
to  establish  either  their  feasibility  or  priority.  The 
problem  was  that  the  requirements  process  was  not  very 
disciplined.  Action  officers  were  asked  to  assemble  lists 
of  requirements  without  any  type  of  restrictions  levied  on 
them.  So,  they've  been  listing  every  possible  item  they 
thought  somebody  might  want  as  part  of  a  new  system.  The 
"nice-to-have"  articles  were  intermingled  with  mission 
essential  elements  and  there  was  never  any  discrimination  of 
what  was  most  important  to  what  was  least  impc  rtant.  As  I 
mentioned  earlier,  this  particular  aspect  of  the 
requirements  process  has  radically  changed  recently  because 
of  the  Chief  of  Staff's  concern  that  this  failure  to 
prioritize  results  in  a  tremendous  price  tag.  For  example, 
we  could  be  buying  a  solid  gold  Cadillac,  when  what  we 
really  need  is  a  very  efficient  Ford  Escort.  Hopefully, 
that's  behind  us.  But  unless  the  four-star  level  continues 
to  oversee  the  results  of  the  initial  requirements  process 
to  ensure  this  prioritization  occurs,  I  believe  we'll  slip 
right  back  into  the  quagmire  we  were  in  before. 

The  second  aspect  of  the  requirements  process  that 
continually  requires  attention  is  the  need  to  ensure  that 
what  is  stated  as  an  operational  requirement  is  not  just  a 
direct  quotation  from  a  specification.  Too  frequently, 
specifications  are  only  piece-part  statements.  Dictating 
that  a  subcomponent  meets  a  certain  mean  time  between 
critical  failures  rate  as  a  "requirement"  ,  is  micromanaging 
and  doesn't  adequately  address  the  mission  requirements. 
Requirements  should  be  stated  in  macro  terms  and  be  more 
mission  oriented. 

These  deficiencies  that  I've  talked  about  have  been 
around  for  a  long  time,  partly  because  it's  difficult  to  do 
what  I've  suggested.  You  know  it's  easy  to  say  what's  wrong 
with  a  process,  but  successfully  changing  it  is  a  tremendous 
challenge.  And,  I'm  not  sure  if  we'll  be  able  to  meet  that 
kind  of  a  challenge. 
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2 .  How  has  AFOTEC  and  the  OT&E  community  responded  to 
inadequate  or  questionable  user  requirements  in  the  past? 

They've  done  two  things  sequentially.  First,  they 
filled  the  void.  If  a  requirement  was  missing  in  an  area  in 
which  AFOTEC  felt  there  should  be  one,  they  created  the 
requirement  and  then  tested  to  it.  This  was  common  until 
about  two  years  ago  when  at  that  time,  AFOTEC  decided  it  was 
inappropriate  for  them  to  do  that.  While  AFOTEC' s  practice 
of  filling  in  the  blanks  may  have  improved  the  requirements 
development  of  the  system,  it  went  against  the  process  of 
developing  a  system's  requirements  because  AFOTEC  isn't  in 
the  business  of  setting  requirements.  AFOTEC  then  took  a 
new  approach  and  decided  they  wouldn't  test  to  anything  if  a 
requirement  wasn't  stated.  Our  objective  was  to  force  the 
user  to  come  to  grips  with  the  fact  that  his  requirements 
process  was  deficient.  Some  good  examples  include  the  B-2, 
C-17  and  MARK  XV.  These  programs  had  to  revise  their 
requirements  because  AFOTEC  refused  to  accept  holes  where 
requirements  should  have  been  and  requirements  that  were 
expressed  inappropriately. 


3.  What  impact  have  the  BCM  and  RCM  had  on  the  definition 
of  user  requirements? 

Although  the  BCM  focused  on  the  fact  that  frequently 
there  were  no  requirements  identified  in  areas  that  were 
critical  to  mission  accomplishment,  its  initial  impact  was 
that  it  drove  the  users  into  becoming  more  micro  in  stating 
subcomponent  capabilities  as  requirements.  They  simply 
looked  at  the  specifications  in  the  contract  and  used  them 
to  fill  in  the  blanks  in  the  BCM.  Even  though  the  BCM 
focused  on  the  problem,  the  user  responded  from  the  wrong 
perspective.  However,  I  think  the  RCM  is  headed  in  the 
right  direction.  It  focuses  on  the  fact  that  there  were 
inadequacies  in  the  requirements  process  and  provides  for 
more  aggregation  of  the  pertinent  operational  aspects  of  the 
system.  The  RCM  is  a  tool  that  can  help  the  user  look  at 
requirements  from  the  right  perspective.  But  the  jury's 
still  out  because  we  haven't  had  that  many  programs  work 
through  the  RCM  process  and  come  out  with  anything  that 
looks  different  from  the  BCM.  So  right  now,  the  RCM  is  the 
BCM  renamed,  and  it'll  take  some  time  to  see  if  anyone  takes 
advantage  of  this  new  tool. 


4.  Is  OSD's  Systems  Maturity  Matrix  a  useful  tool? 

In  a  non-politicized  world  it  would  be  a  useful  tool. 
But  in  a  politicized  world  where  OSD  works,  it  has  the 
potential  to  be  more  destructive  rather  than  constructive. 
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Specifically,  the  maturity  matrix  was  envisioned  as  a  kind 
of  PERT  process  that  indicated  the  amount  of  testing  that 
should  be  accomplished  by  a  certain  point  in  time,  based  on 
how  many  test  articles  were  available  and  the  planned  test 
rate.  It  measures  the  progress  of  the  testing.  The 
problem,  however,  is  that  people  also  want  it  to  measure  how 
well  the  system  is  meeting  the  user's  requirements.  They 
also  want  it  to  predict  how  the  system  will  ultimately 
perform,  based  on  its  present  performance.  That  wasn't  the 
intent;  and,  the  SMM  won’t  give  you  that  type  of 
information.  It's  just  an  inventory  of  test 
accomplishments.  AFOTEC  has  not  yet  been  forced  to  use  the 
SMM  because  it  hasn't  been  formalized.  I'd  like  to  see  the 
demise  of  the  SMM  as  a  tool  because  it  won't  meet  people's 
expectations. 


5.  Did  the  revision  of  AFR  57-1  in  1987-88  lead  to  any 
other  improvements? 

I  don't  know.  The  process  is  still  working  through  its 
first  cycle  and  I'm  not  sure  if  there  will  be  any  side 
benefits  other  than  those  envisioned  to  result  from  the 
revision. 


6.  Has  the  change  in  operational  test  reporting  (meets  or 
does  not  meet  user  requirements)  had  an  impact  on  the 
definition  of  requirements? 

Yes,  it's  had  an  impact  on  what  things  are  now  called 
"requirements".  It  helps  us  delineate  between  what 
information  is  actually  required  and  what  is  information  I 
would  like  to  know.  For  years,  users  have  asked  us,  as  part 
of  our  test  process,  to  accumulate  information  that  will 
allow  us  to  make  a  statement  about  such  things  as  the 
adequacy  of  the  training  syllabus.  That  type  of 
information,  until  recently,  has  been  listed  as  a 
"requirement",  because  we've  been  very  loose  in  our  use  of 
the  word.  Now,  although  we  tell  the  user  whether  we  think 
the  syllabus  is  adequate  to  meet  their  needs,  we  don't 
formally  address  whether  the  training  capability  associated 
with  the  system  meets  or  does  not  meet  user  requirements 
because  we  don't  list  training  as  a  contractual 
"requirement"  anymore.  That  way,  it  doesn't  show  up  in 
Washington  as  a  failure  to  meet  requirements,  although  it 
essentially  is. 

The  other  aspect,  which  is  perhaps  the  single  most 
important  thing  we've  learned,  concerns  the  reporting 
process.  When  someone  uses  terms  that  are  subjective  rather 
than  qualitative,  he  injects  his  own  biases  in  the 
evaluation  and  his  position  becomes  indefensible.  Your 
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position  is  much  more  defensible  when  you  use  a  yardstick 
called  "user  requirements”,  and  those  user  requirements  fall 
someplace  on  that  scale  when  measured  by  that  yardstick. 

It’s  very  difficult  to  argue  with  specifics.  Additionally, 
it's  best  to  avoid  using  terminology  with  a  subjective 
component  like  ”unsatisfactoryM ,  because  these  types  of 
requirements  being  evaluated  are  generally  not  immutable 
absolutes.  Then,  when  someone  says  that  falling  this  far 
below  meeting  the  requirement  is  unsatisfactory,  he's  made  a 
preemptive  statement  and  made  defacto  the  decision. 
Especially,  if  he's  in  Washington,  where  people  fasten  on  to 
words  like  that  and  stop  all  thought  processes  after  hearing 
"unsatisfactory".  Perhaps  that  measurement  was  not  really 
unsatisfactory,  but  just  didn't  meet  the  requirement 
originally  established.  But,  by  using  inappropriate 
evaluation  terminology,  he  preempted  the  decision  process. 
Someone  outside  of  AFOTEC  should  determine  if  the  lower 
measurement  is  a  good  enough  requirement.  That  is  what 
we're  striving  toward. 


7.  What  other  AFOTEC  initiatives  have  helped  improve  the 
process? 

We  do  much  more  extensive  early  planning  so  that  we  have 
carefully  matched  our  test  scenarios  with  the  operational 
concept  and  associated  requirements  to  be  tested.  This 
prevents  us  from  using  too  many  TBDs  right  up  until 
execution  time  and  winging  the  test  activities. 


8.  What  changes  in  requirements  policy  have  come  out  of  the 
recent  "summit  meetings"  on  major  programs  such  as  the  B-l, 
B-2  and  C-17? 

I  don't  think  there's  been  any  change  in  policy 
published  and  I  don't  know  if  there  will  be.  I  suspect  that 
if  there  is  a  policy  change,  it'll  be  an 
institutionalization  of  the  four-star  review  of  these 
programs.  As  of  yet,  however,  there  hasn't  been  a  specific, 
formalized  policy  change. 


9.  Do  you  think  the  Defense  Management  Review  (DMR)  will 
result  in  further  changes? 

Yes,  but  none  of  them  are  relevant  to  the  questions  at 
hand.  The  DMR  is  an  attempt  to  cut  fat  from  the  management 
structure  and  eliminate  duplicative  activities. 
Unfortunately,  it's  so  highly  publicized  that  no  one  knows 
what  the  outcome  will  be.  But  I  don't  think  it's  applicable 
to  the  requirements  process. 
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10.  Will  the  shrinking  defense  budget  affect  the 
requirements  process? 

Not  in  the  context  of  changing  the  structure  or 
development  of  requirements.  But  in  a  pragmatic  sense,  it 
will  probably  force  people  to  be  very  incisive  about  what 
they're  calling  a  requirement  as  opposed  to  developing  a 
long  shopping  list.  It's  just  one  more  pressure  that  says 
someone  can't  just  list  every  possible  thing  he  might  want 
to  have  in  a  system  and  call  it  a  requirement. 


11.  Will  the  DMR  put  even  more  emphasis  on  OT&E? 

I  do  not  think  the  size  of  the  budget  will  have  any 
effect  on  whether  OT&E  receives  emphasis  or  not.  Sadly 
enough,  what  has  increased  emphasis  on  OT&E  is  the  fact  that 
the  OT&E  agencies  are  independent  of  the  major  commands.  I 
think  that  is  an  unfortunate  reason  to  focus  on  OT&E.  It 
implies  that  everyone  in  your  command,  the  user  commands,  or 
anybody  else  who  is  subject  to  some  supervision  below  the 
Chief  of  Staff  level,  does  not  have  any  integrity.  It 
implies  that  because  someone  is  concerned  about  being 
promoted,  he  will  say  whatever  the  program  manager  instructs 
him  to  say.  Unfortunately,  he  is  not  acting  under  the 
pretext  of  being  a  good  soldier  and  following  orders;  he  is 
suppressing  information  that  will  make  the  program  look  bad. 
And  nobody  wants  to  do  that.  There  is  always  some  of  that 
in  any  endeavor,  but  we  do  not  have  fifty  thousand  people  in 
the  acquisition  process  who  are  a  bunch  of  lying,  cheating, 
spineless,  self-aggrandizing  individuals.  But,  because  the 
OT&E  agencies  are  independent  of  everyone  except  the  CNO  and 
Chiefs  of  Staff  of  the  Army  and  Air  Force,  they  are  supposed 
to  be  more  honest.  I  think  it  is  a  damn  shame,  but  that  is 
the  way  it  is.  So,  the  budget  size  will  not  impact  OT&E. 


12.  How  has  congressional  interest/ oversight  most  affected 
OT&E?  .  quirements? 

Congress  believes  contractors,  if  left  to  their  own 
devices,  will  unduly  influence  the  acquisition  process  in 
favor  of  their  product.  But  the  repugnance  with  which 
Congress  views  contractor  involvement  in  the  process  has 
been  taken  to  an  extreme,  because  Congress  claims  any 
contractor  association  with  operational  test  automatically 
provides  this  unwarranted  influence.  What  this  has  done  to 
OT&E  for  Air  Force  systems,  which  tend  to  be  much  more 
technically  complex  than  systems  for  the  other  services,  is 
force  us  to  violate  the  letter  of  the  law.  Why?  Because 
until  you  have  all  the  components  of  a  major  weapon  system, 
such  as  the  special  test  equipment  which  is  delivered  after 
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the  vehicle  is  developed,  all  the  bombs  to  be  used  and 
validated  and  verified  technical  orders,  you  can't 
adequately  test  the  system.  It  is  totally  unreasonable  to 
expect  to  have  all  of  those  things  prior  to  setting  up  the 
full  production  line.  So  then,  we  are  caught  in  a  terrible 
dilemma  due  to  Congressional  stricture  of  contractor 
in/olvement.  We  have  handled  this  by  saying  we  are  meeting 
the  spirit  of  the  law  because  we  do  not  allow  the 
contractors  any  place  at  the  table  during  the  evaluation 
process.  But  sometimes,  we  use  contractor  models  because  it 
would  cost  tens  of  millions  of  dollars  to  establish  the  same 
model;  we  use  their  data  collectors  on  occasion;  and,  we  use 
some  of  their  data  collection  systems  as  well.  As  a  result, 
we  are  violating  the  letter  of  the  law,  but  as  long  as  we  do 
not  allow  the  contractors  to  participate  in  the  evaluation 
process,  we  maintain  that  we  have  kept  sacrosanct  the  spirit 
of  the  law. 


13.  If  you  were  sitting  on  this  side  of  the  table,  are 
there  any  other  questions  you  would  ask  the  AFOTEC 
commander? 

I  would  ask,  "What  is  the  greatest  challenge  you  face  in 
operational  testing."  Discounting  politics,  which  we  can  do 
fairly  easily  because  we  are  insulated  by  the  current  Chief 
of  Staff,  the  greatest  challenge  we  face  is  bei.ig  able  to 
create  and  maintain  for  the  duration  of  the  test,  a  credible 
facsimile  of  a  realistic,  operational  environment.  The 
reason  is  twofold.  First,  the  instrumentation  requirements 
are  expensive  and  must  be  non- intrusive  on  the  outcome  of 
the  test.  Second,  and  more  importantly,  testing  against 
existing  threats,  much  less  against  a  postulated  capability 
that  does  not  yet  exist,  is  a  very  expensive  proposition. 
Unless  our  country  can  do  a  better  job  of  obtaining  those 
threats  that  exist,  we  will  not  be  able  to  adequately 
address  operational  realism.  Building  surrogates  is 
horrendously  expensive,  is  very  time  consuming,  and 
frequently  is  not  effective. 


Final  thoughts: 

If  the  need  expressed  for  a  system  dictates  a  new 
invention,  it,  by  definition,  can  not  be  a  requirement  for 
that  system.  If  someone  makes  so  many  requirements  for  a 
system  such  that  something  new  must  be  invented,  he's  built 
in  automatic  failures  because  no  one  knows  how  the  system 
works.  Requirements  must  be  technically  achievable  at  the 
time  we  embark  on  the  development  of  the  system.  Anything 
beyond  that  falls  into  the  experimental  research  category 
and  should  not  be  associated  with  that  particular 
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acquisition.  This  may  seem  like  a  subtle  point  or  a  play  on 
words,  but  it  is  not.  It  is  a  very  profound  issue.  If  a 
particular  capability  has  been  specified  as  a  requirement, 
but  cannot  be  technically  accomplished,  it  is  no  longer  a 
requirement.  There  is  a  difference  between  what  someone 
says  they  need  and  what  they're  requiring  of  a  particular 
system.  Needs  and  requirements  are  not  synonymous,  even 
though  they’re  frequently  translated  that  way. 


Interview  conducted  by:  Captain  David  Struck,  AFIT/LSG 

Mr.  Lawrence  Benson,  AFOTEC/RS 
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Appendix  B:  Lessons  Learned 


USAF  Lessons  Learned  Impact  Areas 

1.  Computer  Resources  (Support) 

2 .  Energy  Management 

3.  Engineering  Data  (Technical  Data) 

4.  Facilities 

5.  Funding  (Logistics) 

6.  Logistics  Management  Information  Support 

7.  Maintainability 

8.  Maintenance  Concept  (Planning) 

9.  Modification  Planning 

10.  Manpower  Requirements 

11.  Reliability 

12.  Reliability  &  Maintainability 

13.  Safety 

14 .  Supply  Support 

15.  Support  Equipment 

16.  Survivability 

17.  Technical  Orders  (Technical  Data) 

18.  Test  and  Evaluation 

19.  Transportation  Packaging  and  Handling 

20.  Training  and  Training  Support 

21.  Artificial  Intelligence 

22.  Propulsion  Systems 

23.  Systems  Integration  (Hardware) 

24.  Systems  Integration  (Software) 

25.  Software 

26.  Software  Management 

27.  Configuration  Management 

28.  Contract  Administration 

29.  Contracting 

30.  Data  Management 

31.  Engineering 

32.  Foreign  Military  Sales 

33.  Human  Factors  Engineering 

34.  Life  Cycle  Cost 

35.  Manufacturing 

36.  Operational  Requirements 

37.  Program  Control 

38.  Quality  Assurance 

39.  Source  Selection 

40.  Program  Management  Responsibility  Transfer 

41.  Logistics  Support  Analysis 

42.  Program  Management 

43.  Environmental  Management 

44.  Warranties 

45.  Hazardous  Materials 

46.  Automated  Information  Systems 
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47.  Total  Quality  Management  (TQM) 

48.  Personnel 

49.  Operations 


USAF  Lessons  Learned  Abstract 

CALL  NUMBER:  2109  VALIDATOR:  AFALC/ERRT/S CHUCK 

TOPIC:  EFFECTIVE  TEST  PLANNING  (REQUIREMENTS  CORRELATION 

MATRIX) 

LESSON  LEARNED:  Test  program  objectives  must  be  correlated 
to  either  operational  or  clearly  defined  developmental 
requirements.  Failure  to  do  so  will  result  in  test  time 
devoted  towards  objectives  which  serve  no  useful  purpose. 
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OT&E  LESSONS  LEARNED  REPORT 


PROGRAM  NAME/TYPE  OF  TEST:  AUTOMATED  REMOTE  TRACKING 
STATION  (ARTS) /IOT&E 

PHASE  OF  TEST  DURING  WHICH  LESSON  WAS  LEARNED:  TEST 
EXECUTION 

DATE:  16  AUG  1989  AFOTEC-LL-89-3 19 

OFFICIAL/OFFICE  SUBMITTING  REPORT:  AFOTEC  -  DET  4 

FOCUS  OF  LESSON  LEARNED:  MANAGEMENT 

TOPIC  OF  LESSON:  BASELINE  CORRELATION  MATRIX  (BCM) 
REQUIREMENTS  PROCESS 

LESSON:  Don't  rush  the  BCM  process  just  to  get  the  test 

underway;  the  results  are  poorly  defined  and/or 
inappropriate  user  requirements  and  test  criteria. 

DISCUSSION:  The  BCM  for  the  ARTS  Acquisition  I  test  program 

was  apparently  worked  in  a  hurry  to  support  the  IOT&E 
schedule.  Throughout  the  test  program,  various  members  of 
the  ARTS  community  held  different  interpretations  of  the 
user's  requirements  and,  therefore,  the  IOT&E  test  criteria, 
as  well. 

For  example,  the  user's  requirement  for  "mission  success 
rate"  was  identified  in  the  BCM  as  a  mature  requirement; 
however,  "mature"  was  not  defined  until  the  Test  and 
Evaluation  Master  Plan  (TEMP)  was  approved,  which  was 
virtually  at  the  end  of  the  test  program.  The  term  "mature" 
was  defined  as  one  year  after  IOT&E,  with  no  threshold 
provided  for  IOT&E.  Therefore,  by  definition,  we  never 
tested  against  the  user's  actual  requirement.  As  a 
testimony  to  the  validity  or  importance  of  this  requirement, 
we  were  not  asked  to  take  advantage  of  the  one  opportunity 
we  had  to  evaluate  a  "mature"  capability. 

Also,  once  the  BCM  was  signed  it  was  as  if  it  were  in 
concrete.  Although  it  was  obvious  that  there  were  problems 
with  the  interpretations  of  the  user's  requirements,  there 
was  a  reluctance  to  bring  up  the  subject  "because  changes 
would  have  to  be  staffed  back  up  to  the  generals,  and  that 
took  time,  and  we  didn't  have  the  time." 

SOLUTION:  Either  start  the  BCM  process  earlier  or  postpone 

testing. 
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Appendix  C:  Cover  Letter  for  Questionnaires 


LSY 

Research  Questionnaire  on  Requirements 


1.  You  have  been  selected  to  participate  in  an  Air  Force 
research  project  which  is  important  in  the  evaluation  of  Air 
Force  weapon  systems.  The  responses  you  give  to  the 
attached  research  questions  will  be  used  as  a  data  source 
for  determining  the  effectiveness  of  the  Requirements 
Correlation  Matrix  (RCM)  and  Baseline  Correlation  Matrix 

( BCM) .  Since  you  are  a  key  player  in  the  identification  of 
the  parameters  and  values  contained  in  these  documents,  your 
input  is  extremely  valuable. 

2 .  Please  take  a  few  minutes  to  provide  this  important 
information.  Please  answer  each  question  as  accurately  and 
truthfully  as  possible.  All  responses  will  be  held 
confidential,  and  no  attempt  will  be  made  to  match  any 
specific  individual  with  specific  responses.  Of  course, 
your  participation  is  strictly  voluntary. 

3.  Please  return  your  completed  questionnaire  (through 
distribution)  in  the  envelope  provided  within  one  week  of 
receipt.  Any  questions  concerning  this  questionnaire  should 
be  directed  to  Captain  Dave  Struck,  AFIT/LSG,  AUTOVON 
785-8989.  Your  help  in  this  important  project  is  greatly 
appreciated. 


JOHN  DUMOND,  Lt  Col ,  USAF 
Head,  Department  of  System 
Acquisition  Management 
School  of  Systems  and  Logistics 


2  Atch 

1.  Questionnaire 

2.  Return  Envelope 
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uestionnaire  Part  I 
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Part  I  of  AFOTEC  Questionnaire 

RESEARCH  QUESTIONNAIRE  ON 
AIR  FORCE  REQUIREMENTS 

The  following  research  questionnaire  has  been  developed 
to  solicit  information  on  Air  Force  requirements  as  stated 
in  a  Requirements  Correlation  Matrix  (RCM)  or  Baseline 
Correlation  Matrix  (BCM) .  Your  inputs  will  be  included  in 
an  AFIT  thesis  which  will  examine  the  effectiveness  of  the 
RCM  and  BCM. 

Your  participation  in  this  research  effort  is  voluntary 
and  anonymous;  however,  your  cooperation  is  appreciated  and 
will  directly  impact  this  research  effort. 

PART  I 

INSTRUCTIONS 

Please  provide  answers  to  the  following  questions  in  the 
space  provided.  Your  answers  should  be  complete  but 
concise;  however,  feel  free  to  include  any  additional 
information  that  you  feel  is  relevant.  If  you  need  more 
space  to  answer  a  question,  continue  on  the  reverse  side  of 
the  page  on  which  the  question  is  found.  If  you  have  any 
additional  comments  after  completing  this  questionnaire, 
please  write  them  on  the  back  of  the  last  page  of  the 
questionnaire. 


1.  What  is  your  current  AFSC  and  rank/grade?  Are  you 
rated? 


2.  What  is  your  present  position  title  and  what  duties 
(briefly)  does  this  position  entail? 


3.  How  long  have  you  been  assigned  to  the  AFOTEC  Plans  and 
Policy  Directorate? 
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4.  What  were  your  previous  assignments  (identify  base. 
Command,  organization,  length  of  tour,  and  AFSC  for  each 
position  held)  and  what  type  of  duties  did  you  perform? 


5.  How  do  you  define  a  requirement? 


6.  How  do  you  define  a  specification? 


7.  What  do  you  use  as  a  basis  for  these  definitions  (PCE 
classes,  personal  experience,  etc)?  Be  specific. 


8.  Do  you  provide  assistance  in  the  selection  of  parameters 
and/or  values  to  the  developers  of  SONs,  SORDs,  or  RCM/BCMs? 
If  yes,  what  level  of  involvement  do  you  have? 
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Part  I  of  Using  Command  Questionnaire 


RESEARCH  QUESTIONNAIRE  ON 
AIR  FORCE  REQUIREMENTS 

The  following  research  questionnaire  has  been  developed 
to  solicit  information  on  Air  Force  requirements  as  stated 
in  a  Requirements  Correlation  Matrix  (RCM)  or  Baseline 
Correlation  Matrix  ( BCM) .  Your  inputs  will  be  included  in 
an  AFIT  thesis  which  will  examine  the  effectiveness  of  the 
RCM  and  BCM. 

Your  participation  in  this  research  effort  is  voluntary 
and  anonymous;  however,  your  cooperation  is  appreciated  and 
will  directly  impact  this  research  effort. 

PART  I 


INSTRUCTIONS 

Please  provide  answers  to  the  following  questions  in  the 
space  provided.  Your  answers  should  be  complete  but 
concise;  however,  feel  free  to  include  any  additional 
information  that  you  feel  is  relevant.  If  you  need  more 
space  to  answer  a  question,  continue  on  the  reverse  side  of 
the  page  on  which  the  question  is  found.  If  you  have  any 
additional  comments  after  completing  this  questionnaire, 
please  write  them  on  the  back  of  the  last  page  of  the 
quest ionna ire . 


1.  What  is  your  current  AFSC  and  rank/grade?  Are  you 
rated? 


2.  What  is  your  present  position  title  and  what  duties 
(briefly)  does  this  position  entail? 


3.  How  long  have  you  been  assigned  to  the  DCS/Requirements 
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4.  What  were  your  previous  assignments  (identify  base, 
Command,  organization,  length  of  tour,  and  AFSC  for  each 
position  held)  and  what  type  of  duties  did  you  perform? 


5.  How  do  you  define  a  requirement? 


6.  How  do  you  define  a  specification? 


7.  What  do  you  use  as  a  basis  for  these  definitions  (PCE 
classes,  personal  experience,  etc)?  Be  specific. 


8.  Are  you  currently  involved  in  the  development  of 
requirements  for  SONs,  SORDs ,  or  RCM/BCMs? 


9.  If  the  answer  to  the  previous  question  was  yes,  do  you 
seek  assistance  from  individual.-,  or  organizations  outside  of 
your  office?  If  so,  from  whom? 
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Part  I  of  ASD  Questionnaire 


RESEARCH  QUESTIONNAIRE  ON 
AIR  FORCE  REQUIREMENTS 

The  following  research  questionnaire  has  been  developed 
to  solicit  information  on  Air  Force  requirements  as  stated 
in  a  Requirements  Correlation  Matrix  (RCM)  or  Baseline 
Correlation  Matrix  (BCM) .  Your  inputs  will  be  included  in 
an  AFIT  thesis  which  will  examine  the  effectiveness  of  the 
RCM  and  BCM. 

Your  participation  in  this  research  effort  is  voluntary 
and  anonymous;  however,  your  cooperation  is  appreciated  and 
will  directly  impact  this  research  effort. 

PART  I 


INSTRUCTIONS 

Please  provide  answers  to  the  following  questions  in  the 
space  provided.  Your  answers  should  be  complete  but 
concise;  however,  feel  free  to  include  any  additional 
information  that  you  feel  is  relevant.  If  you  need  more 
space  to  answer  a  question,  continue  on  the  reverse  side  of 
the  page  on  which  the  question  is  found.  If  you  have  any 
additional  comments  after  completing  this  questionnaire, 
please  write  them  on  the  back  of  the  last  page  of  the 
questionnaire. 


1.  What  is  your  current  AFSC  and  rank/grade?  Are  you 
rated? 


2.  What  is  your  present  position  title  and  what  duties 
(briefly)  does  this  position  entail? 


3.  How  long  have  you  been  assigned  to  the  ASD  Deputy  for 
Development  Planning? 
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4.  What  were  your  previous  assignments  (identify  base, 
Command,  organization,  length  of  tour,  and  AFSC  for  each 
position  held)  and  what  type  of  duties  did  you  perform? 


5.  How  do  you  define  a  requirement? 


6.  How  do  you  define  a  specification? 


7.  What  do  you  use  as  a  basis  for  these  definitions  (PCE 
classes,  personal  experience,  etc)?  Be  specific. 


8.  Do  you  provide  assistance  in  the  selection  of  parameters 
and/or  values  to  the  developers  of  RCM/BCMs?  If  yes,  what 
level  of  involvement  do  you  have? 
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3.  Speed  at  sea  level  600  KTAS 


4.  Takeoff  distance  9,100  feet 

450,000  lb  GW 


5.  Threat/systexns 
survivability 


Highest  practical 
level  of  protection 
against  12.7  mm 
armor  piercing 
incendiary 


6.  Sortie  generation 
rate  (peacetime) 


1.05 


PARAMETER 


CAT. 


VALUE 


CAT. 


7.  Cruise  speed/initial 
cruise  altitude 


.87  mach/ 
25,500  ft 


8.  A/A  weapons  employ¬ 
ment,  AIM-9 


AIM-9  can  be 
employed  to  full 
a/c  limits 


9.  Conventional  weapon 
delivery — laser 
guided  bomb 


60%  delivery 
success  rate 


10.  Takeoff/landing 
performance 


Takeoff  in  75%  of 
available  runway 


11.  Ferry  range 


3500  nm 


12.  A/A  weapons  employ¬ 
ment,  AIM-9 


A/C  system  supports 
employment  of  AIM-9 


13.  Break  rate 


Maximum  14.3%  of 
sorties 


14.  Useful  life 


18  years 


15.  Radar  A/G  resolution 


13  ft  out  to  20  NM 


Please  return  the  completed  questionnaire  in  the  attached 
envelope.  Thank  you  for  vour  participation. 
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Appendix  F:  Responses  to  Question  5 


How  do  you  define  a  requirement? 

AFOTEC : 

1.  System-specific  operational  capability  necessary  to 
satisfy  an  operational  mission  deficiency. 

2.  A  characteristic  or  function  that  must  be  obtained  to 
accomplish  a  specific  mission. 

3.  Those  elements  essential  to  successfully  improving  and 
completing  the  mission. 

4.  An  outcome  which  must  be  accomplished  in  order  to 
successfully  carry  out  the  designated  mission. 

5.  The  desired  characteristics  of  a  new/ improved  system. 

6.  The  need  that  must  be  met,  given  a  set  of  conditions. 

7.  A  mission  a  system  must  perform  or  a  capability  it  must 
have . 


MAC: 

1.  A  general  statement  (qualitative  or  quantitative)  of  a 
needed  capability. 

2.  A  need  for  something  new. 

3.  In  the  acquisition  process,  a  requirement  is  usually 
thought  of  as  expressing  a  using  command's  need.  That  need 
or  requirement  could  be  the  result  of  a  deficiency  in  a 
current  system,  a  changing  threat,  or  an  application  of  new 
technology  to  provide  operational  requirements. 

4.  A  requirement  is  usually  spelled  out  in  a  SON  and 
further  defined  in  a  SORD.  It  states  the  user's  needs  and 
continues  throughout  the  acquisition  process.  The  key  word 
is  need. 

5.  A  need,  necessity  which  is  indispensable  to  meeting  the 
mission  of  the  weapon  system. 

6.  A  requirement  defines  a  user  need  that  must  be  met  in 
order  for  the  "thing"  being  acquired  to  perform  its  intended 
mission.  A  requirement  should  be  written  in  operational 
terms . 
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7.  Contained  in  SORD/SOC.  Tested  in  IOT&E,  QOT&E.  What 
the  user  needs  to  go  to  war  broken  down  into  operational 
effectiveness  and  suitability  issues  by  critical  issues, 
objectives,  MOE,  and  evaluation  criteria. 

8.  A  need. 

9.  The  end  user’s  operational  need. 


SAC: 

1.  A  need  written  in  operational  terms  which  can  be 
evaluated/confirmed. 

2.  Widget  needed  to  improve  the  warfighting  capability, 
maintainability  or  reliability  of  a  system  or  new 
capability. 

3.  Requirement  is  a  need  or  a  ’’must  have." 

4.  Requirement  is  a  needed  capability  (needed  to  accomplish 
assigned  military  mission) .  It  is  the  result  of  an 
identified  shortfall  which  can  not  be  resolved  by  a  change 
in  strategy,  tactics,  doctrine,  or  training. 

5.  A  mission  capability. 

6.  A  need  established  by  the  user  and  coordinated  with  an 
analysis  of  the  threat. 

7.  Requirement  is  a  definition  of  a  user  need  (i.e.,  need 
is  met  by  satisfaction  of  requirements) .  Requirement 
further  defined  by  a  derived  requirement  (i.e.,  derived 
requirement  further  defines  the  general  requirement)  more 
specific. 

8.  A  need  or  necessity  which  must  be  satisfied  in  order  to 
fulfill  a  condition. 

9.  What  is  necessary  to  accomplish  stated  needs. 

10.  A  statement  of  what  must  be  done.  Should  not  say  how 
it  must  be  done.  May  be  general  or  specific.  Typically, 
prepared  by  a  user  prior  to  contracting  stage  of  procurement 
process.  A  requirement  becomes  a  goal  when  it  exceeds 
minimum  capability  for  mission. 

11.  Formalization  and  validation  of  a  mission  need 
generated  by  the  operational  community. 
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12.  Analysis  of  changing  threat,  or  change  in  tactics  or 
mission,  technological  obsolescence,  unsupportability 
problems,  reduced  reliability  of  the  system.  May  be 
surfaced  through  material  improvement  (MIP)  from  the  field 
or  depot,  or  a  recurring  problem  surfaced  from  an  operator. 

13.  A  requirement  is  the  iterative  statement  that  develops 
from  a  basic  deficiency  in  capability.  Stated  in 
operational  terms,  a  need  is  refined  through  a  disciplined 
process  of  review  and  evaluation  to  state  the  data  on  which 
a  solution  can  be  based.  Technological  opportunity  may  also 
generate  a  requirement. 

14.  A  requirement  represents  the  translation  of  an 
operational  need  into  characteristics  that  the  system  must 
have  for  the  intended  mission.  The  operational  need  is 
derived  from  employment,  logistic,  manpower,  and  training 
constraints  and  opportunities  associated  with  stated 
operational  objectives  and  can  be  met  through  changes  in 
tactics,  strategy,  doctrine,  training,  new  development,  new 
procurement,  or  upgrade  of  an  existing  system.  In  my 
opinion,  the  term  "requirement”  is  the  most  abused  term  in 
the  acquisition  process.  In  AFR  57-1  and  throughout  the 
texts  for  Defense  Systems  Management  College,  there  are 
operational,  test,  design,  performance,  system,  item,  input, 
functional,  reporting,  intelligence,  and  mission 
requirements.  This  term's  overuse  to  describe  the 
parameters,  cl.aracteristics,  design,  plan,  outcomes, 
standards,  features,  traits,  components,  elements, 
attributes,  etc.  of  a  system  dilutes  the  significance  of  the 
operational  need.  The  only  relevant  requirements  are  the 
operational  needs  of  the  warfighting  commanders  or  the 
commanders  supporting  the  war fighting  commanders.  These 
requirements  can  not  always  be  quantified  or  measured  (i.e., 
stop  the  advance  of  heavy  armor,  deny  the  use  of  forward 
operating  locations,  perform  deep  interdiction  of  critical 
transportation  nodes,  extend  the  life  expectancy  of  crews  in 
combat) .  Requirements  are  seldom  precise  and  do  not 
necessarily  describe  a  particular  solution 


TAC: 

1.  Operators'  needs  to  allow  them  to  complete  the  mission. 

2.  An  approved,  recognized  need  for  some  capability. 

3.  What  you  need  to  do  your  mission. 

4.  A  required  (necessary)  operational  capability;  major 
system  (e.g.  F-15E) ,  sub-system  (e.g.  APG-70  radar). 
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5.  An  Air  Force  user  operational  need. 

6.  A  requirement  is  a  generic  solution  to  an  operational 
need,  i.e.,  a  night  vision  device  which  can  detect  and 
identify  tanks. 

7.  When  the  operations  community  determines  what  it  needs  to 
do  to  counter  threats;  the  equipment,  software,  systems, 
etc.,  that  can  accomplish  that  mission  becomes  a 
requirement. 


ASD: 

1.  A  mission  capability,  need-derived  system-level 
capability. 

2.  Requirements  are  defined  in  terms  of  operational  jobs 
that  need  to  be  accomplished.  This  needs  to  be  described  in 
means  of  operational  measures  (tons  in  a  time  period,  size 
of  vehicle  transported,  runway  length  restrictions) . 

3.  An  operational  deficiency  which  can  be  met  with  a  system 
solution.  A  requirement  implies  that  the  system  solution  is 
affordable,  feasible,  and  can  be  built  within  a  known 
schedule . 

4.  It  is  an  expression  of  need  by  a  using  command  for  a  new 
capability. 

5.  A  need  for  a  functional  capability. 

6.  A  requirement  is  a  need  established  by  the  user.  A 
requirement  is  necessary  for  the  user  to  accomplish  his 
mission. 

7.  Need  to  perform  the  user’s  mission  and  a  force 
multiplier. 

8.  Requirements  are  essential  needs  and  objectives  to  any 
concept  that  will  yield  a  product  or  tangible  service. 
Requirements  are  simply  building  blocks  and  foundations. 

9.  Something  needed  to  perform  a  particular  function. 

10.  System  operational  capability  needed  to  fulfill  an 
operational  objective. 

11.  A  military  capability  that  is  necessary  to  accomplish  a 
defined  mission. 

12.  A  system  that  fulfills  a  user  need  or  deficiency. 
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13.  I  have  difficulties  with  the  word  "requirement.” 
Basically,  a  requirement  for  new  systems,  or  mods  to 
existing  systems,  must  be  established  by  the  combatant 
command  (SAC,  NORAD,  USAFE,  etc.)  to  support  operational 
objectives  (provide  close  air  support,  defeat  enemy  air 
attacks,  etc.). 

14.  Established  by  using  command  to  quantify  need,  often 
expressed  as  baseline  and  goal. 
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Appendix  G:  Responses  to  Question  6 


How  do  you  define  a  specification? 

AFOTEC: 

1.  System  specific  design  criteria  specified  in  a  contract 
which,  if  satisfied,  should  contribute  to  meeting  a  system’s 
required  operational  capability. 

2.  A  testable,  verifiable  characteristic  or  function 
contractually  agreed  to  by  buyer  and  contractor  and  derived 
from  requirements. 

3.  Those  elements  that  make  up  the  engineering  and  design 
of  a  system  to  fulfill  the  developmental  requirement. 

4.  A  parameter,  derived  from  operational  requirements, 
delineating  required  system  performance  and  which  is 
contractually  described  and  binding  on  both  the  government 
and  the  vendor  (usually  quantifiable) . 

5.  A  quantitative  value  used  to  define  a  contractual 
requirement. 

6.  A  detailed  requirement. 

7.  A  contractual  value  from  which  an  item  can  be  built;  a 
measure  of  a  physical  character ' "tie . 


MAC: 


1.  A  specific  statement  (qualitative  or  quantitative)  of  a 
needed  capability  that  can  be  measured. 

2.  The  things  an  item  should  conform  to  or  do. 

3.  A  specification  is  a  contract  document  between  the 
implementing  command  and  contractor  that  spells  out 
specifically  the  performance  requirements  (ops  and  maint)  of 
the  acquired  system.  It  is  derived  from  the  using  command 1 s 
requirements  documents  and  PMD. 

4.  A  specification  is  basically  a  statement  of  particulars 
to  identify  exactly  what/or  how  an  item  should  look  or  work 
in  terms  of  form,  fit  and  function. 

5.  A  spec  or  series  of  specs  are  technical 
requirements/sub-requirements  normally  accomplished  within 
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the  acquisition  communities  to  meet  validated  user 
requirements. 

6.  A  specification  is  a  contractual  parameter  that  the 
■'thing"  must  meet.  If  the  "thing"  meets  the  specification, 
then  the  user's  requirements  should  also  be  met. 

7.  Contained  in  spec/statement  of  work,  tested  in 
DT&E/QT&E;  what  the  contractor  must  meet  to  get  paid. 

8.  A  description  of  a  need. 

9.  That  which  should  meet  the  requirement. 


SAC: 

1.  A  detailed  task(s)  which  can  be  tested/ evaluated. 

2.  The  required  documentation  to  ensure  the  requirement  is 
met . 

3.  A  detailed  description  of  requirements. 

4.  Specification  is  a  contractually  binding  performance 
level  identified  for  various  parameters  of  a  solution  for  a 
validated  requirement. 

5.  A  performance  requirement. 

6.  Supporting  criteria  within  the  requirement.  The 
requirement  specifies  particular  support  items,  e.g.,  a 
nuclear  hardness  criteria  is  a  specification  within  a 
requirement  for  nuclear  hardness. 

7.  Specification  quantifies  a  requirement  so  that  a 
contractor  can  build  to  spec  thus  fulfilling  the 
requirement . 

8.  A  definite  and  complete  description  of  requirements  in 
an  organized  fashion  clearly  stated. 

9.  Clear  description  of  a  certain  item. 

10.  The  translation  of  a  requirement  into  a  contract 
RFP/f inalized  contract.  Must  be  specific  and  verifiable.  A 
proper  specification  is  not  overly  constrained,  i.e.,  top 
speed  greater  than  600  knots  vice  top  speed  of  600  knots. 
Specifications  may  change  during  the  course  of  an 
acquisition. 
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11.  Detailed  description  of  a  system's  dimensions, 
capabilities  or  composition. 

12.  We  do  not  write  specifications.  We  write  requirements 
that  come  from  field  problems  or  future  threats.  The  SPO  or 
AFLC  (ACC)  turns  our  requirements  into  specifications  and 
statements  of  work  for  contractual  efforts. 

13.  Contractual  measurement  on  which  a  deliverable  item  can 
be  assessed.  Spec  is  derived  from  requirement  and  must 
ensure  that  the  basic  need  is  met. 

14.  The  distinction  between  a  "requirement"  and  a 
"specification"  is  the  degree  of  precision  by  which 
specifications  describe  a  system.  The  term  "requirement" 
has  not  been  well  defined  within  the  acquisition  process, 
however,  the  term  "specification"  has  been  strictly  defined. 
Requirements  from  need  statements  are  analyzed  in  the 
systems  engineering  process  and  translated  into  system  and 
developmental  specifications  which  ensure  that  requirements 
are  properly  stated  and  traceable  within  the  configuration 
management  process.  Specifications  are  prepared  to  a 
standard  format  (Mil  Std  490A)  and  terminology  and  establish 
requirements  in  terms  of  complete  design  details  and 
performance.  "General"  specifications  apply  to  all  programs 
and  incorporate  many  government  standards  which  define 
items,  approaches,  procedures,  or  testing  to  be  used  in 
development  and  production  processes.  "Program-peculiar" 
are  specifications  that  define  unique  mission  needs. 


TAC: 

1.  Weight,  dimensions,  capabilities. 

2.  The  exact  definition  of  the  guidelines  and/or 
limitations  needed  to  describe/meet  a  requirement. 

3.  The  parameters  of  what  is  needed  to  do  the  mission; 
shape,  size,  range,  speed,  etc. 

4.  An  ingredient/s  that  make  up  a  solution  to  fulfill  the 
requirement,  reliability,  supportable,  and  within  cost 
parameters . 

5.  System  operating  parameter  which  will  fulfill  the 
requirement.  A  quantifiable  point  which  can  be  evaluated 
during  Test  and  Evaluation. 

6.  An  engineering  definition  of  a  requirement. 
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7.  Specific  characteristic  of  a  system  which  is  a  solution 
to  the  user  requirement,  i.e.,  8-12  mission  FLIR  with  10 
degrees  field  of  view. 

8.  When  a  requirement  has  been  established  and  put  on 

contract,  then  the  requirement  has  become  a  specification. 
Problem:  requirements  change  faster  than  specifications. 


ASD: 


1.  A  system-level  capability  developed  to  the  level  of 
detail  needed  to  develop,  design  and  manufacture 
hardware/software . 

2.  Specifications  define  a  system  and  how  it  must  look  and 
perform  to  meet  the  requirements.  Specifications  are 
derived  to  make  the  system  perform  as  required. 

3 .  It  is  a  measure  of  performance  which  the  system  must 
meet. 

4.  It  is  a  detailed  description  of  an  item  which  describes 
the  actual  or  required  size,  performance,  quality,  terms  and 
other  particulars  that  will  provide  for  its  development  and 
production. 

5.  A  specific  attempt  to  address  a  requirement. 

6.  System  specification  is  a  measurable  end  item  of  a 
hardware  contract. 

7.  Bounds  placed  on  equipment;  what  it's  to  do;  size, 
weight,  affordability  and  practibility . 

8.  Specifications  imply  that  you  shall  determine  the 
precision  of  a  given  article.  Specifications  describe  how 
items  will  be  built  or  orders  carried  out. 

9.  Detailed  explanation  of  how  to  meet  a  requirement. 

10.  Those  attributes  of  a  system  required  to  meet  a  system 
level  operational  capability. 

11.  An  in-depth  description  with  detailed  characteristics 
of  a  desired  physical  item. 

12.  Document  that  details  the  configuration  and  performance 
aspects  of  a  system. 
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13.  The  value  of  a  specific  performance  parameter;  e.g. 
speed,  range  detection  distance,  etc.  In  other  words, 
factors  that  can  be  controlled  by  the  designer. 

14.  Contract  direction;  "build  to";  must  be  measurable. 
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Appendix  H:  Sources  of  RCM  Assistance 


SAC: 

HQ  SAC/XR/XRF/XRRR/XRRM/DO/LG/XO/XP/IN/DE/DP/IG 

HQ  USAF/XOORE 

HQ  SAC/SMOD 

Contractor  engineers 

OC-ALC  (engineering  and  management) 

ASD  (engineering,  m  lgement  and  costing) 

SPO 

Testers 


MAC: 

Contractors  ("other  government  agencies  often  provide  more 
help  than  I  need") 

Equipment  operators 
Using  commands 
ASD  SPO  engineering 
HQ  MAC  staff 
HQ  AFOTEC/TEZ 
Previous  requirements 
Implementing  commands 


TAC: 

HQ  TAC  staff  (/DO  and  /LG) 

Air  Staff 

Coordination  process 
AFSC  Product  Divisions 
AFOTEC 
AFLC 

Senior  leadership  within  commands 
Operating  units 

"Poor  response  from  coordinating  agencies" 
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Appendix  I:  Categorizations  for  All  Respondents 
Parameter  Categorizations  (%) 


QUESTION 

R 

S 

B 

N 

1 

12.2 

36.7 

20.4 

30.6 

2 

38.8 

8.2 

24.5 

28 . 6 

3 

20.4 

22.4 

22.4 

34.7 

4 

28 . 6 

24.5 

20.4 

26.5 

5 

46.9 

4.1 

12.2 

36.7 

6 

53.1 

2 . 0 

10.2 

34.7 

7 

32.7 

26.5 

12.2 

28.6 

8 

46.9 

2 . 0 

12.2 

38.8 

9 

51.0 

6.1 

10.2 

32.7 

10 

26.5 

8 . 2 

20.4 

44.9 

11 

38.8 

4.1 

16.3 

40.8 

12 

40.8 

6.1 

10.2 

42.9 

13 

26.5 

12.2 

16.3 

44.9 

14 

34.7 

10.2 

12.2 

42.9 

15 

30.6 

14.3 

20.4 

34.7 

Value  Categorizations  (%) 


QUESTION 

R 

S 

B 

N 

1 

4.1 

65.3 

22.4 

8.2 

2 

20.4 

36.7 

32.7 

10.2 

3 

18.4 

36.7 

36.7 

8.2 

4 

16.3 

49.0 

26.5 

8.2 

5 

42.9 

18.4 

18.4 

20.4 

6 

42 . 9 

14.3 

30.6 

12.2 

7 

14 . 3 

49.0 

26.5 

10.2 

8 

49.0 

20.4 

12.2 

18.4 

9 

40.8 

22.4 

20.4 

16.3 

10 

32.7 

14.3 

18.4 

34.7 

11 

34.7 

30.6 

26.5 

8.2 

12 

51.0 

10.2 

18.4 

20.4 

13 

28.6 

28 . 6 

20.4 

22.4 

14 

36.7 

26.5 

24.5 

12.2 

15 

12.2 

55.1 

22.4 

10.2 
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Appendix  J:  Categorizations  bv  Aeronautical  Rating 


Rated  Parameter  Categorizations  (%) 


QUESTION 

R 

S 

B 

N 

1 

10.0 

40.0 

20.0 

30.0 

2 

35.0 

5.0 

30.0 

30.0 

3 

15.0 

30.0 

20.0 

35.0 

4 

25.0 

35.0 

10.0 

30.0 

5 

35.0 

5.0 

15.0 

45.0 

6 

45.0 

5.0 

10.0 

40.0 

7 

20.0 

25.0 

20.0 

35.0 

8 

50.0 

5.0 

10.0 

35.0 

9 

45.0 

10.0 

15.0 

30.0 

10 

25.0 

15.0 

15.0 

45.0 

11 

30.0 

15.0 

5.0 

50.0 

12 

35.0 

10.0 

10.0 

45.0 

13 

20.0 

10.0 

25.0 

45.0 

14 

20.0 

10.0 

15.0 

55.0 

15 

25.0 

30.0 

20.0 

25.0 

Non-Rated  Parameter  Categorizations  (%) 


QUESTION 

R 

S 

B 

N 

1 

13.8 

34 . 5 

20.7 

31.0 

2 

41.4 

10.3 

20.7 

27.6 

3 

24.1 

17.2 

24 . 1 

34.5 

4 

31.0 

17.2 

27.6 

24.1 

5 

55.2 

3.4 

10.3 

31.0 

6 

58.6 

0.0 

10.3 

31.0 

7 

37.9 

27.6 

10.3 

24.1 

8 

44.8 

0.0 

13.8 

41.4 

9 

55.2 

3.4 

6.9 

34.5 

10 

27.6 

3.4 

24.1 

44.8 

11 

41.4 

3.4 

20.7 

34.5 

12 

44.8 

3.4 

10.3 

41.4 

13 

31.0 

13.8 

10.3 

44.8 

14 

44.8 

10.3 

10.3 

34.5 

15 

34.5 

3.4 

20.7 

41.4 
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Rated  Value  Categorizations  (%) 


OUESTION 

R 

£ 

B 

N 

1 

0.0 

75.0 

20.0 

5.0 

2 

30.0 

30.0 

35.0 

5.0 

3 

20.0 

45.0 

30.0 

5.0 

4 

20.0 

50.0 

25.0 

5.0 

5 

30.0 

25.0 

20.0 

25.0 

6 

55.0 

15.0 

25.0 

5.0 

7 

10.0 

55.0 

25.0 

10.0 

8 

60.0 

20.0 

10.0 

10.0 

9 

40.0 

20.0 

25.0 

15.0 

10 

30.0 

15.0 

20.0 

35.0 

11 

35.0 

35.0 

25.0 

5.0 

12 

65.0 

10.0 

15.0 

10.0 

13 

30.0 

25.0 

30.0 

15.0 

14 

25.0 

35.0 

30.0 

10.0 

15 

15.0 

70.0 

15.0 

0.0 

Non- 

■Rated  Value  Categorizations 

(%) 

OUESTION 

E 

£ 

£ 

N 

1 

6.9 

58.6 

24.1 

10.3 

2 

13.8 

41.4 

31.0 

13.8 

3 

17.2 

31.0 

41.4 

10.3 

4 

13.8 

48.3 

27.6 

10.3 

5 

51.7 

10.3 

20.7 

17.2 

6 

37.9 

10.3 

34.5 

17.2 

7 

13.8 

48.3 

27.6 

10.3 

8 

41.4 

20.7 

13.8 

24.1 

9 

41.4 

24 . 1 

17.2 

17.2 

10 

34.5 

13.8 

17.2 

34.5 

11 

31.0 

27.6 

31.0 

10.3 

12 

41.4 

10.3 

20.7 

27.6 

13 

27.6 

31.0 

13.8 

27.6 

14 

44.8 

20.7 

20.7 

13.8 

15 

10.3 

48.3 

24.1 

17.2 
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Appendix  K;  Categorizations  bv  Organization 


Using  Command  Parameters 


« 


\ 


QUESTION 

R 

S 

B 

N 

1 

17.9 

28.6 

25.0 

28.6 

2 

35.7 

10.7 

17.9 

35.7 

3 

21.4 

10.7 

32 . 1 

35.7 

4 

25.0 

21.4 

25.0 

28.6 

5 

46.4 

3.6 

10.7 

39.3 

6 

39.3 

0.0 

14.3 

46.4 

7 

32.1 

17.9 

17.9 

32.1 

8 

42 . 9 

0.0 

3 . 6 

53.6 

9 

53 . 6 

0.0 

7.1 

39.3 

10 

25.0 

7.1 

17.9 

50.0 

11 

42.9 

3.6 

10.7 

42.9 

12 

39.3 

3 . 6 

3.6 

53.6 

13 

25.0 

7.1 

14.3 

53.6 

14 

39.3 

3.6 

10.7 

46.4 

15 

32.1 

7.1 

17.9 

42.9 

Using 

Command  Values 

QUESTION 

R 

s 

B 

N 

1 

7.1 

64.3 

25.0 

3.6 

2 

21.4 

39.3 

32.1 

7.1 

3 

21.4 

28.6 

46.4 

3.6 

4 

17.9 

39.3 

39.3 

3.6 

5 

46.4 

14 . 3 

25.0 

14.3 

6 

42 . 9 

7.1 

39.3 

10.7 

7 

21.4 

39.3 

35.7 

3.6 

8 

50.0 

17.9 

14.3 

17.9 

9 

50.0 

21.4 

21.4 

7.1 

10 

39.3 

10.7 

25.0 

25.0 

11 

46.4 

17.9 

35.7 

0.0 

12 

53.6 

7.1 

17.9 

21.4 

13 

32.1 

25.0 

25.0 

17.9 

14 

42.9 

21.4 

28.6 

7.1 

15 

17.9 

50.0 

25.0 

7.1 
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ASD/XR  Parameters 


QUESTION 

R 

S 

B 

N 

1 

7.1 

42.9 

7.1 

42.9 

2 

50.0 

7.1 

21.4 

21.4 

3 

28.6 

14.3 

14.3 

42.9 

4 

42.9 

14.3 

14.3 

28.6 

5 

42.9 

7.1 

7.1 

42.9 

6 

64.3 

7 . 1 

7.1 

21.4 

7 

42.9 

28.6 

0.0 

28.6 

8 

50.0 

7.1 

21.4 

21.4 

9 

42.9 

21.4 

7.1 

28.6 

10 

28.6 

14.3 

14.3 

42.9 

11 

28.6 

7 . 1 

21.4 

42.9 

12 

50.0 

14.3 

14.3 

21.4 

13 

21.4 

21.4 

14.3 

42.9 

14 

28 . 6 

28 . 6 

7.1 

35.7 

15 

21.4 

28.6 

21.4 

28.6 

ASD/XR  Values 


QUESTION 

R 

S 

B 

N 

1 

0.0 

64.3 

21.4 

14.3 

2 

21.4 

35.7 

28.6 

14.3 

3 

21.4 

42.9 

21.4 

14 . 3 

4 

21.4 

57.1 

7.1 

14.3 

5 

50.0 

14 . 3 

7.1 

28.6 

6 

35.7 

28.6 

21.4 

14 . 3 

7 

7.1 

57.1 

14.3 

21.4 

8 

50.0 

28 . 6 

0.0 

21.4 

9 

35.7 

28.6 

7.1 

28.6 

10 

28.6 

21.4 

0.0 

50.0 

11 

21.4 

42.9 

14.3 

21.4 

12 

50.0 

21.4 

14.3 

14.3 

13 

28.6 

28.6 

14.3 

28.6 

14 

35.7 

28.6 

14.3 

21.4 

15 

7.1 

57.1 

21.4 

14.3 

4 
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AFOTEC  Parameters 


QUESTION 

R 

S 

B 

N 

1 

0.0 

57 . 1 

28.6 

14.3 

2 

28.6 

0.0 

57.1 

14.3 

3 

0.0 

85.7 

0.0 

14.3 

4 

14.3 

57.1 

14.3 

14.3 

5 

57.1 

0.0 

28.6 

14.3 

6 

85.7 

0.0 

0.0 

14.3 

7 

14.3 

57.1 

14.3 

14.3 

8 

57.1 

0.0 

28.6 

14.3 

9 

57.1 

0.0 

28.6 

14.3 

10 

28.6 

0.0 

42.9 

28.6 

11 

42.9 

0.0 

28.6 

28.6 

12 

28.6 

0.0 

28.6 

42.9 

13 

42.9 

14 . 3 

28.6 

14.3 

14 

28 . 6 

0.0 

28.6 

42.9 

15 

42.9 

14.3 

28.6 

1'  .3 

AFOTEC 

Values 

QUESTION 

R 

S 

B 

N 

1 

0.0 

71.4 

14.3 

14.3 

2 

14.3 

28.6 

42.9 

14.3 

3 

0.0 

57.1 

28.6 

14 . 3 

4 

0.0 

71.4 

14.3 

14.3 

5 

14.3 

42.9 

14.3 

28.6 

6 

57.1 

14.3 

14.3 

14.3 

7 

0.0 

71.4 

14.3 

14.3 

8 

42.9 

14.3 

28.6 

14.3 

9 

14 . 3 

14 . 3 

42.9 

28.6 

10 

14 . 3 

14.3 

28.6 

42.9 

11 

14.3 

57.1 

14.3 

14.3 

12 

42.9 

0.0 

28.6 

28.6 

13 

14.3 

42.9 

14.3 

28.6 

14 

14 . 3 

42.9 

28.6 

14.3 

15 

0.0 

71.4 

14.3 

14.3 
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Appendix  L:  Categorizations  by  Experience  Level 


Parameters 

Less  Than  2  Years  Experience 


QUESTION 

R 

S 

B 

N 

1 

11.5 

46.2 

23.1 

19.2 

2 

42.3 

11.5 

19.2 

26.9 

3 

15.4 

26.9 

26.9 

30.8 

4 

38 . 5 

23.1 

19.2 

19.2 

5 

46.2 

7.7 

15.4 

30.8 

6 

50.0 

3.8 

7.7 

38.5 

7 

30.8 

30.8 

15.4 

23.1 

8 

42.3 

3.8 

11.5 

42.3 

9 

53.8 

3.8 

15.4 

26.9 

10 

23 . 1 

11.5 

26.9 

38.5 

11 

42 . 3 

3.8 

19.2 

34 . 6 

12 

34.6 

7.7 

11.5 

46.2 

13 

26.9 

15.4 

15.4 

42.3 

14 

42.3 

7.7 

11.5 

38.5 

15 

34.6 

15.4 

15.4 

34.6 

Less  Than 

Values 
2  Years 

Experience 

QUESTION 

R 

S 

B 

N 

1 

3 . 8 

76.9 

15.4 

3.8 

2 

23.1 

53.8 

15.4 

7.7 

3 

23.1 

38.5 

34.6 

3.8 

4 

15.4 

57.7 

23.1 

3.8 

5 

50.0 

23 . 1 

15.4 

11.5 

6 

42.3 

19.2 

6.9 

11.5 

7 

15.4 

53.8 

26.9 

3.8 

8 

46.2 

26.9 

7.7 

19.2 

9 

42.3 

26.9 

19.2 

11.5 

10 

38.5 

19.2 

15.4 

26.9 

11 

46.2 

34 . 6 

19.2 

0.0 

12 

53.8 

11.5 

11.5 

23.1 

13 

34.6 

34 . 6 

19.2 

11.5 

14 

50.0 

30.8 

11.5 

7.7 

15 

11.5 

65.4 

15.4 

7.7 
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Parameters 


More 

than  2  but 

Less  than  4 

Years 

Experience 

QUESTION 

R 

S 

B 

N 

1 

14.3 

21.4 

28.6 

35.7 

2 

35.7 

7.1 

28.6 

28.6 

3 

21.4 

21.4 

21.4 

35.7 

4 

14.3 

28.6 

28.6 

28.6 

5 

50.0 

0.0 

7.1 

42.9 

6 

57.1 

0.0 

14.3 

28.6 

7 

35.7 

21.4 

14.3 

28.6 

8 

50.0 

0.0 

14.3 

35.7 

9 

42.9 

7.1 

7.1 

42.9 

10 

35.7 

7.1 

7.1 

50.0 

11 

28.6 

7.1 

14.3 

50.0 

12 

42.9 

0.0 

14.3 

42.9 

13 

28.6 

0.0 

21.4 

50.0 

14 

21.4 

7 . 1 

21.4 

50.0 

15 

28.6 

14.3 

28.6 

28.6 

More 

than  2  but 

Values 
Less  than  4 

Years 

Experience 

QUESTION 

R 

S 

B 

N 

1 

7.1 

57.1 

35.7 

0.0 

2 

28.6 

14 . 3 

57.1 

0.0 

3 

21.4 

35.7 

42.9 

0.0 

4 

21.4 

42.9 

35.7 

0.0 

5 

35.7 

21.4 

21.4 

21.4 

6 

64.3 

7.1 

28.6 

0.0 

7 

14.3 

57.1 

21.4 

7.1 

8 

64.3 

14.3 

21.4 

0.0 

9 

50.0 

21.4 

21.4 

7.1 

10 

21.4 

21.4 

28.6 

28.6 

11 

14.3 

28.6 

50.0 

7.1 

12 

50.0 

7.1 

35.7 

7.1 

13 

28.6 

28.6 

21.4 

21.4 

14 

21.4 

28.6 

42.9 

7.1 

15 

21.4 

57.1 

21.4 

0.0 
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Parameters 

More  than  4  Years  Experience 


QUESTION 

R 

S 

B 

N 

1 

11.1 

33 . 3 

0.0 

55.6 

2 

33.3 

0.0 

33.3 

33.3 

3 

33 . 3 

11.1 

11.1 

44.4 

4 

22.2 

22 . 2 

11.1 

44.4 

5 

44.4 

0.0 

11.1 

44.4 

6 

55.6 

0.0 

11.1 

33.3 

7 

22.2 

22 . 2 

11.1 

44.4 

8 

55.6 

0.0 

11.1 

33.3 

9 

66.7 

11.1 

0.0 

22.2 

10 

22.2 

0.0 

22.2 

55.6 

11 

33.3 

0.0 

22.2 

44.4 

12 

55.6 

11.1 

0.0 

33.3 

13 

22.2 

22.2 

11.1 

44.4 

14 

33.3 

22.2 

0.0 

44.4 

15 

22.2 

11.1 

22.2 

44.4 

More  than 

Values 
4  Years 

Experience 

QUESTION 

R 

S 

B 

N 

1 

0.0 

44.4 

22.2 

33.3 

2 

0.0 

22.2 

44.4 

33.3 

3 

0.0 

33.3 

33.3 

33.3 

4 

11.1 

33.3 

22.2 

33.3 

5 

33 . 3 

0.0 

22.2 

44.4 

6 

22.2 

11.1 

33.3 

33.3 

7 

0.0 

33 . 3 

33.3 

33.3 

8 

33 . 3 

11.1 

11.1 

44.4 

9 

22.2 

11.1 

22.2 

44.4 

10 

22.2 

0.0 

11.1 

66.7 

11 

11.1 

22 . 2 

33.3 

33.3 

12 

44.4 

11.1 

11.1 

33.3 

13 

11.1 

11.1 

22.2 

55.6 

14 

22.2 

11.1 

33.3 

33.3 

15 

0.0 

22.2 

44.4 

33.3 

/ 
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